Sep 10 14:42:08.139982 ip-10-0-141-186 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 14:42:08.571567 ip-10-0-141-186 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:42:08.571567 ip-10-0-141-186 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 14:42:08.571567 ip-10-0-141-186 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:42:08.571567 ip-10-0-141-186 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 14:42:08.571567 ip-10-0-141-186 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 14:42:08.574116 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.574023    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 14:42:08.578393 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578373    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:42:08.578393 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578390    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:42:08.578393 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578395    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:42:08.578393 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578399    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578404    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578408    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578411    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578415    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578420    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578424    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578428    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578432    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578437    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578440    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578444    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578448    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578452    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578456    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578460    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578464    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578468    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578472    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578476    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:42:08.578640 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578480    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578484    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578488    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578492    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578496    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578512    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578517    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578522    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578526    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578530    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578535    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578539    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578543    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578547    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578553    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578557    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578561    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578566    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578571    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578575    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:42:08.579472 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578579    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578583    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578598    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578603    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578607    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578611    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578617    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578621    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578626    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578630    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578634    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578638    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578643    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578647    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578651    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578655    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578660    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578664    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578684    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578688    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:42:08.580329 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578692    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578697    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578701    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578706    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578710    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578714    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578720    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578726    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578730    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578734    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578738    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578743    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578747    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578751    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578755    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578760    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578764    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578772    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578779    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:42:08.581171 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578783    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578788    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578793    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578798    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578804    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578810    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578816    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578823    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578827    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578832    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578837    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578841    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578848    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578852    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578857    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578861    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.578865    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580018    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580028    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:42:08.581995 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580033    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580038    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580043    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580047    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580052    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580056    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580060    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580064    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580068    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580072    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580077    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580081    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580085    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580089    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580093    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580097    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580101    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580105    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580109    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580113    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:42:08.582837 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580117    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580121    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580125    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580130    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580134    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580139    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580144    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580148    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580152    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580156    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580160    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580164    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580168    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580172    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580176    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580181    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580185    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580189    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580193    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580197    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580201    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:42:08.583526 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580205    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580209    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580214    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580218    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580222    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580226    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580230    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580234    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580238    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580242    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580246    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580251    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580255    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580259    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580263    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580270    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580274    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580278    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580285    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:42:08.584098 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580291    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580296    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580300    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580304    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580308    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580312    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580316    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580320    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580324    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580328    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580335    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580339    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580344    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580348    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580352    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580356    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580360    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580364    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580369    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580373    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:42:08.584565 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580377    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580381    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580385    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580389    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580394    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580402    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580409    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580415    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580420    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580425    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580429    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580434    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580440    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580445    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580450    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580454    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.580459    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581063    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:42:08.585243 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581071    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581076    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581080    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581085    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581092    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581097    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581101    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581105    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581109    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581114    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581118    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581122    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581126    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581130    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581135    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581140    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581144    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581148    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581152    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581157    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:42:08.585758 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581161    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581165    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581169    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581172    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581176    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581182    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581189    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581194    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581199    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581203    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581208    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581213    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581217    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581221    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581226    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581230    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581235    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581239    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581243    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:42:08.586245 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581248    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581252    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581256    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581260    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581265    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581269    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581273    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581279    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581283    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581287    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581292    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581296    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581300    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581304    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581308    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581312    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581316    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581320    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581325    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581330    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:42:08.586823 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581336    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581340    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581344    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581348    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581352    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581356    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581361    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581365    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581370    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581375    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581380    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581384    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581388    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581392    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581396    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581401    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581404    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581408    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581413    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581419    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:42:08.587350 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581423    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581427    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581434    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581440    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581444    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581448    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581452    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581457    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581461    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581466    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581470    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581474    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581478    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581484    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581489    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581495    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581500    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581504    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.581508    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581621    2563 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 14:42:08.587850 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581685    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581696    2563 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581703    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581709    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581714    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581721    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581727    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581732    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581737    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581743    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581748    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581755    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581760    2563 flags.go:64] FLAG: --cgroup-root=""
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581765    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581770    2563 flags.go:64] FLAG: --client-ca-file=""
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581775    2563 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581779    2563 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581785    2563 flags.go:64] FLAG: --cluster-domain=""
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581789    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581794    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581800    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581806    2563 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581812    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 14:42:08.588341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581816    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581821    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581826    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581832    2563 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581837    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581842    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581847    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581852    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581858    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581863    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581868    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581872    2563 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581877    2563 flags.go:64] FLAG: --enable-server="true"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581882    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581889    2563 flags.go:64] FLAG: --event-burst="100"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581894    2563 flags.go:64] FLAG: --event-qps="50"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581898    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581903    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581908    2563 flags.go:64] FLAG: --eviction-hard=""
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581914    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581921    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581926    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581931    2563 flags.go:64] FLAG: --eviction-soft=""
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581935    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581940    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 14:42:08.588951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581945    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581949    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581954    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581958    2563 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581963    2563 flags.go:64] FLAG: --feature-gates=""
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581970    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581975    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581980    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581985    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581990    2563 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581994    2563 flags.go:64] FLAG: --help="false"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.581999    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582004    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582009    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582013    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582019    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582027    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582032    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582037    2563 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582042    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582047    2563 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582051    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582056    2563 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582061    2563 flags.go:64] FLAG: --kube-reserved=""
Sep 10 14:42:08.589630 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582066    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582070    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582075    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582080    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582085    2563 flags.go:64] FLAG: --lock-file=""
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582089    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582093    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582098    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582106    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582111    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582116    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582121    2563 flags.go:64] FLAG: --logging-format="text"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582125    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582131    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582135    2563 flags.go:64] FLAG: --manifest-url=""
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582140    2563 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582147    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582151    2563 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582158    2563 flags.go:64] FLAG: --max-pods="110"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582163    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582168    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582174    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582178    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582183    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582189    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 14:42:08.590556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582194    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582205    2563 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582210    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582215    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582220    2563 flags.go:64] FLAG: --pod-cidr=""
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582225    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582243    2563 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582248    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582253    2563 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582258    2563 flags.go:64] FLAG: --port="10250"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582263    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582268    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d11c93c1b6e51512"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582273    2563 flags.go:64] FLAG: --qos-reserved=""
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582278    2563 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582283    2563 flags.go:64] FLAG: --register-node="true"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582288    2563 flags.go:64] FLAG: --register-with-taints=""
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582300    2563 flags.go:64] FLAG: --registry-burst="10"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582305    2563 flags.go:64] FLAG: --registry-qps="5"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582309    2563 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582314    2563 flags.go:64] FLAG: --reserved-memory=""
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582320    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582325    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582330    2563 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582334    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582339    2563 flags.go:64] FLAG: --runonce="false"
Sep 10 14:42:08.591442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582344    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582349    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582354    2563 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582359    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582365    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582369    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582377    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582381    2563 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582386    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582391    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582395    2563 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582400    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582405    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582410    2563 flags.go:64] FLAG: --system-cgroups=""
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582415    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582424    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582429    2563 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582433    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582439    2563 flags.go:64] FLAG: --tls-min-version=""
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582443    2563 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582447    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582452    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582457    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582462    2563 flags.go:64] FLAG: --v="2"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582468    2563 flags.go:64] FLAG: --version="false"
Sep 10 14:42:08.592104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582475    2563 flags.go:64] FLAG: --vmodule=""
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582480    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.582486    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582698    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582706    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582711    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582715    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582719    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582723    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582727    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582731    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582736    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582742    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582750    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582754    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582759    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582763    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582767    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582771    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582775    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:42:08.592767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582779    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582784    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582788    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582792    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582796    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582800    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582804    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582808    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582812    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582816    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582820    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582825    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582829    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582833    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582836    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582840    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582844    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582848    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582852    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582857    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:42:08.593263 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582861    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582865    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582869    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582873    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582878    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582884    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582889    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582893    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582897    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582903    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582907    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582912    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582916    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582920    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582924    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582928    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582932    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582936    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582940    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:42:08.593767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582945    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582952    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582958    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582962    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582967    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582972    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582976    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582980    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582984    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582988    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582992    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.582997    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583001    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583005    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583009    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583013    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583017    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583021    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583027    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583031    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:42:08.594238 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583035    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583039    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583045    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583049    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583056    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583062    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583066    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583071    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583075    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583080    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583084    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583088    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583092    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583096    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583100    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583105    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583109    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583113    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583117    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:42:08.594752 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583121    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583125    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583129    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.583133    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.583141    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.592527    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.592543    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.593528    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.593536    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593590    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593595    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593599    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593602    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593605    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593607    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:42:08.595222 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593610    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593613    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593616    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593619    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593622    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593625    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593627    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593630    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593632    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593635    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593638    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593642    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593646    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593649    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593652    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593655    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593657    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593660    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593663    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:42:08.595606 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593666    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593683    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593686    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593695    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593699    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593702    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593704    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593707    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593710    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593714    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593717    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593720    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593723    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593726    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593728    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593731    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593734    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593736    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593739    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:42:08.596128 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593741    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593744    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593747    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593751    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593754    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593756    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593759    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593762    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593764    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593767    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593769    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593773    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593775    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593778    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593781    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593783    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593786    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593789    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593792    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:42:08.596725 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593794    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593797    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593800    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593802    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593805    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593808    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593810    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593813    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593816    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593818    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593821    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593823    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593825    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593828    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593830    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593833    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593837    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593840    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593842    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593845    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:42:08.597190 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593847    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593849    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593852    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593854    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593857    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593859    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593862    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593865    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593867    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593870    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593872    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593875    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593878    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593880    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593883    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:42:08.597689 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.593885    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.593890    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594003    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594009    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594012    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594015    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594017    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594020    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594023    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594026    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594028    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594031    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594033    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594036    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 14:42:08.598057 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594039    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594042    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594044    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594047    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594050    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594052    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594055    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594057    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594060    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594062    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594065    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594067    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594070    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594072    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594075    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594077    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594080    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594083    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594085    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 14:42:08.598397 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594088    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594090    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594093    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594095    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594099    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594103    2563 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594106    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594109    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594112    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594114    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594117    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594119    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594122    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594124    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594127    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594130    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594132    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594135    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594137    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 14:42:08.598891 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594140    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594142    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594145    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594147    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594150    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594152    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594155    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594158    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594160    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594163    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594165    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594168    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594170    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594173    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594176    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594178    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594181    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594183    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594187    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 14:42:08.599351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594190    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594193    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594196    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594199    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594201    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594204    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594206    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594209    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594211    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594215    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594217    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594220    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594223    2563 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594225    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594229    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594233    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594236    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594238    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594241    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594244    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 14:42:08.599920 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594247    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594250    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594253    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594255    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594258    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594260    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594263    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594266    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594268    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:08.594270    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.594275    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.594375    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.596797    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.597885    2563 server.go:1019] "Starting client certificate rotation"
Sep 10 14:42:08.600406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.597987    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:42:08.600778 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.598871    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 14:42:08.601315 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.601301    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:42:08.603400 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.603383    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 14:42:08.611742 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.611726    2563 log.go:25] "Validated CRI v1 runtime API"
Sep 10 14:42:08.614967 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.614952    2563 log.go:25] "Validated CRI v1 image API"
Sep 10 14:42:08.615891 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.615877    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 14:42:08.620311 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.620288    2563 fs.go:135] Filesystem UUIDs: map[1fe390b4-95b3-47c7-8f4e-30ca97085ebb:/dev/nvme0n1p3 23bd0ce5-c908-4197-abe4-f5c49a347dc0:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 10 14:42:08.620374 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.620307    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 14:42:08.627407 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.627295    2563 manager.go:216] Machine: {Timestamp:2026-09-10 14:42:08.625817462 +0000 UTC m=+0.372713001 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102203 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2239f8c7e20b5e01919827f9f11635 SystemUUID:ec2239f8-c7e2-0b5e-0191-9827f9f11635 BootID:66c97763-861a-4fff-ac5e-6811984fb421 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a2:ba:1c:78:3d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a2:ba:1c:78:3d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:8e:d6:87:a0:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 14:42:08.627407 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.627395    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 14:42:08.627521 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.627476    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 14:42:08.630442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.630426    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:42:08.630994 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.630970    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 14:42:08.631123 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.630998    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-186.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 14:42:08.632534 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.632524    2563 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 14:42:08.632569 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.632536    2563 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 14:42:08.632569 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.632544    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:42:08.633390 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.633380    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 14:42:08.635291 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.635280    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 14:42:08.639004 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.638992    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:42:08.639109 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.639101    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 14:42:08.639149 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.639130    2563 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 14:42:08.639149 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.639145    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 14:42:08.639203 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.639163    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 14:42:08.639203 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.639177    2563 kubelet.go:398] "Adding apiserver pod source"
Sep 10 14:42:08.639203 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.639189    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 14:42:08.639595 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.639582    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:42:08.639648 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.639605    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 14:42:08.642045 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.642026    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 14:42:08.645033 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645016    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 14:42:08.645119 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645055    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 14:42:08.645246 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645234    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 14:42:08.645303 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645252    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 14:42:08.645303 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645260    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 14:42:08.645303 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645268    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 14:42:08.645303 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645280    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 14:42:08.645303 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645290    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 14:42:08.645868 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645856    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 14:42:08.645921 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645870    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 14:42:08.645921 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645881    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 14:42:08.645921 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645891    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 14:42:08.645921 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645914    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 14:42:08.646051 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.645928    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 14:42:08.646865 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.646853    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 14:42:08.646924 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.646867    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 14:42:08.648421 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.648403    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-186.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 14:42:08.648527 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.648504    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-186.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 14:42:08.648623 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.648604    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 14:42:08.650740 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.650726    2563 server.go:1268] "Started kubelet"
Sep 10 14:42:08.650820 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.650794    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 14:42:08.650866 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.650800    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 14:42:08.650866 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.650863    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 14:42:08.651654 ip-10-0-141-186 systemd[1]: Started Kubernetes Kubelet.
Sep 10 14:42:08.652346 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.652329    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 14:42:08.653579 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.653566    2563 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 14:42:08.656581 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.655896    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 14:42:08.657033 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.655860    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-186.ec2.internal.18d3fcb4612a7961  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-186.ec2.internal,UID:ip-10-0-141-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-186.ec2.internal,},FirstTimestamp:2026-09-10 14:42:08.650705249 +0000 UTC m=+0.397600788,LastTimestamp:2026-09-10 14:42:08.650705249 +0000 UTC m=+0.397600788,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-186.ec2.internal,}"
Sep 10 14:42:08.658632 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.658615    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 14:42:08.659307 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.659294    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 14:42:08.659431 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.659413    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:08.660111 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660085    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 14:42:08.660199 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660087    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 14:42:08.660199 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660124    2563 factory.go:55] Registering systemd factory
Sep 10 14:42:08.660199 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660161    2563 factory.go:223] Registration of the systemd container factory successfully
Sep 10 14:42:08.660199 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660125    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 14:42:08.660395 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660249    2563 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 14:42:08.660447 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660397    2563 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 14:42:08.660447 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660370    2563 factory.go:153] Registering CRI-O factory
Sep 10 14:42:08.660447 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660421    2563 factory.go:223] Registration of the crio container factory successfully
Sep 10 14:42:08.660573 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660485    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 14:42:08.660573 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660505    2563 factory.go:103] Registering Raw factory
Sep 10 14:42:08.660573 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660522    2563 manager.go:1196] Started watching for new ooms in manager
Sep 10 14:42:08.660871 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.660858    2563 manager.go:349] Starting recovery of all containers
Sep 10 14:42:08.672808 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.672778    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-186.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 14:42:08.672910 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.672883    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 14:42:08.672968 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.672916    2563 manager.go:354] Recovery completed
Sep 10 14:42:08.677382 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.677367    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:42:08.679171 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.679147    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:42:08.679248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.679184    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:42:08.679248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.679194    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:42:08.679761 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.679743    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 14:42:08.679761 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.679761    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 14:42:08.679867 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.679778    2563 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 14:42:08.679867 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.679778    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:42:08.680527 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.680506    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:42:08.680591 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.680536    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:42:08.680591 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.680549    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:42:08.680791 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.680777    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-97n5n"
Sep 10 14:42:08.681139 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.681125    2563 policy_none.go:49] "None policy: Start"
Sep 10 14:42:08.681184 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.681141    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 14:42:08.681184 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.681151    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 14:42:08.682300 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.682285    2563 policy_none.go:47] "Start"
Sep 10 14:42:08.687382 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.687304    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-186.ec2.internal.18d3fcb462dcd6ba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-186.ec2.internal,UID:ip-10-0-141-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-186.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-186.ec2.internal,},FirstTimestamp:2026-09-10 14:42:08.67917177 +0000 UTC m=+0.426067313,LastTimestamp:2026-09-10 14:42:08.67917177 +0000 UTC m=+0.426067313,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-186.ec2.internal,}"
Sep 10 14:42:08.688250 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.688233    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-97n5n"
Sep 10 14:42:08.689102 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.689037    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-186.ec2.internal.18d3fcb462dd18d8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-186.ec2.internal,UID:ip-10-0-141-186.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-186.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-186.ec2.internal,},FirstTimestamp:2026-09-10 14:42:08.679188696 +0000 UTC m=+0.426084239,LastTimestamp:2026-09-10 14:42:08.679188696 +0000 UTC m=+0.426084239,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-186.ec2.internal,}"
Sep 10 14:42:08.714248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.714234    2563 manager.go:337] "Starting Device Plugin manager"
Sep 10 14:42:08.725692 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.714274    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 14:42:08.725692 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.714287    2563 server.go:87] "Starting device plugin registration server"
Sep 10 14:42:08.725692 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.714600    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 14:42:08.725692 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.714614    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 14:42:08.725692 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.714728    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 14:42:08.725692 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.714813    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 14:42:08.725692 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.714824    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 14:42:08.725692 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.715542    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 14:42:08.725692 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.715597    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:08.754321 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.754293    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 14:42:08.755502 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.755486    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 14:42:08.755565 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.755530    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 14:42:08.755565 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.755551    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 14:42:08.755653 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.755588    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 14:42:08.758070 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.758047    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:42:08.815159 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.815133    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:42:08.816095 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.816079    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:42:08.816160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.816112    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:42:08.816160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.816135    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:42:08.816222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.816166    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.825709 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.825629    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.825709 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.825651    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-186.ec2.internal\": node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:08.848988 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.848971    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:08.855973 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.855956    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal"]
Sep 10 14:42:08.856035 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.856012    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:42:08.857198 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.857183    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:42:08.857271 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.857218    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:42:08.857271 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.857227    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:42:08.858508 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.858497    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:42:08.858610 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.858596    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.858648 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.858624    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:42:08.859197 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.859180    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:42:08.859282 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.859213    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:42:08.859282 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.859223    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:42:08.859282 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.859186    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:42:08.859423 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.859290    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:42:08.859423 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.859309    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:42:08.860450 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.860435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.860525 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.860458    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 14:42:08.861077 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.861057    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/143d3343a56237b77a570303f223bfcd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal\" (UID: \"143d3343a56237b77a570303f223bfcd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.861168 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.861083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/143d3343a56237b77a570303f223bfcd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal\" (UID: \"143d3343a56237b77a570303f223bfcd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.861168 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.861099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8d04f9a962d39a0595afd41d5bd3b7a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-186.ec2.internal\" (UID: \"c8d04f9a962d39a0595afd41d5bd3b7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.861779 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.861762    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 14:42:08.861836 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.861790    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 14:42:08.861836 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.861799    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeHasSufficientPID"
Sep 10 14:42:08.888262 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.888239    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-186.ec2.internal\" not found" node="ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.892485 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.892468    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-186.ec2.internal\" not found" node="ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.949619 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:08.949602    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:08.961403 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.961379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/143d3343a56237b77a570303f223bfcd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal\" (UID: \"143d3343a56237b77a570303f223bfcd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.961403 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.961404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/143d3343a56237b77a570303f223bfcd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal\" (UID: \"143d3343a56237b77a570303f223bfcd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.961547 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.961420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8d04f9a962d39a0595afd41d5bd3b7a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-186.ec2.internal\" (UID: \"c8d04f9a962d39a0595afd41d5bd3b7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.961547 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.961444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c8d04f9a962d39a0595afd41d5bd3b7a-config\") pod \"kube-apiserver-proxy-ip-10-0-141-186.ec2.internal\" (UID: \"c8d04f9a962d39a0595afd41d5bd3b7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.961547 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.961475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/143d3343a56237b77a570303f223bfcd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal\" (UID: \"143d3343a56237b77a570303f223bfcd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:08.961547 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:08.961482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/143d3343a56237b77a570303f223bfcd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal\" (UID: \"143d3343a56237b77a570303f223bfcd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:09.050604 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:09.050560    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:09.151379 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:09.151307    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:09.190738 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:09.190708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:09.194819 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:09.194748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:09.252176 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:09.252143    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:09.352678 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:09.352643    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:10.723053 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:10.723025    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:10.823616 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:10.823581    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-186.ec2.internal\" not found"
Sep 10 14:42:10.837259 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.837240    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:42:10.863290 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.863055    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:42:10.867404 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.867387    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 14:42:10.867556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.867536    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:42:10.867618 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.867553    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:42:10.867618 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.867532    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 14:42:10.909884 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.909857    2563 apiserver.go:52] "Watching apiserver"
Sep 10 14:42:10.925583 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.925557    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 14:42:10.926607 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.926586    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-g9xjj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn","openshift-cluster-node-tuning-operator/tuned-w2rd5","openshift-multus/multus-additional-cni-plugins-d9dr7","openshift-multus/multus-vzpmc","openshift-multus/network-metrics-daemon-lk75l","kube-system/konnectivity-agent-ggpnx","openshift-dns/node-resolver-25clg","openshift-image-registry/node-ca-c4tpn","openshift-network-diagnostics/network-check-target-mdn58","openshift-network-operator/iptables-alerter-6p8sc","openshift-ovn-kubernetes/ovnkube-node-kfmvq"]
Sep 10 14:42:10.928100 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.928084    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:10.928181 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:10.928159    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:10.929218 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.929201    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 14:42:10.929361 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.929345    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:10.929466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.929452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:10.930575 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.930558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.931294 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.931276    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 14:42:10.931401 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.931372    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 14:42:10.931498 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.931483    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 14:42:10.931572 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.931538    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7t6xn\""
Sep 10 14:42:10.931841 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.931827    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:10.932523 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.932475    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:42:10.932637 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.932611    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 14:42:10.932692 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.932657    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qvtnm\""
Sep 10 14:42:10.933075 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.933029    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:10.933409 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.933392    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 14:42:10.933595 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.933580    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 14:42:10.933817 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.933802    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 14:42:10.933920 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.933907    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q2tx4\""
Sep 10 14:42:10.933975 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.933936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 14:42:10.933975 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.933946    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 14:42:10.934323 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.934309    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:10.934403 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:10.934371    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:10.934754 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.934731    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 14:42:10.934837 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.934778    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wsft2\""
Sep 10 14:42:10.935107 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935091    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-kubernetes\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935143 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-run\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935143 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-lib-modules\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935213 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0020ef57-1c96-4818-a96e-6f545c206074-tmp\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935213 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-kubelet-config\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:10.935290 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935210    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-socket-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:10.935290 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:10.935290 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-sysctl-conf\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46grl\" (UniqueName: \"kubernetes.io/projected/0020ef57-1c96-4818-a96e-6f545c206074-kube-api-access-46grl\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:10.935406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:10.935406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rjn\" (UniqueName: \"kubernetes.io/projected/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-kube-api-access-49rjn\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:10.935406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-host\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-sysconfig\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-sys\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935450    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-var-lib-kubelet\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0020ef57-1c96-4818-a96e-6f545c206074-etc-tuned\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-dbus\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-sys-fs\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935541    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-modprobe-d\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935554    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-sysctl-d\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-systemd\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-registration-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:10.935611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-device-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:10.936045 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.935774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:10.936945 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.936930    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:10.937411 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.937395    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 14:42:10.937411 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.937405    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 14:42:10.937532 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.937488    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hlwsw\""
Sep 10 14:42:10.938209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.938196    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:10.938285 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.938211    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 14:42:10.938609 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.938595    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 14:42:10.938677 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.938622    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 14:42:10.938736 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.938720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9p9w6\""
Sep 10 14:42:10.939577 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.939560    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:10.939664 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:10.939609    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:10.939797 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.939784    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 14:42:10.940063 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.940040    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 14:42:10.940132 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.940042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qhscm\""
Sep 10 14:42:10.940132 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.940077    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 14:42:10.942748 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.942728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:10.944011 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.943995    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:42:10.944088 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.944059    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal"
Sep 10 14:42:10.944144 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.944095    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:10.944222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.944203    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal"]
Sep 10 14:42:10.944582 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.944550    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:42:10.944759 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.944745    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 14:42:10.944896 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.944868    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 14:42:10.945934 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.945921    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 14:42:10.945934 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.945930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 14:42:10.946146 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.946131    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 14:42:10.946224 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.946160    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jqkh4\""
Sep 10 14:42:10.946300 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.946252    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jxjhx\""
Sep 10 14:42:10.946443 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.946430    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 14:42:10.946597 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.946578    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 14:42:10.946700 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.946668    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 14:42:10.952368 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.952354    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal"]
Sep 10 14:42:10.952454 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.952442    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 14:42:10.960058 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.960014    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 14:37:09 +0000 UTC" deadline="2028-03-08 06:08:41.313095284 +0000 UTC"
Sep 10 14:42:10.960124 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.960072    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13071h26m30.353039483s"
Sep 10 14:42:10.968079 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.968063    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-spcj5"
Sep 10 14:42:10.976669 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:10.976651    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-spcj5"
Sep 10 14:42:11.031239 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.031224    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 14:42:11.035737 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbd7\" (UniqueName: \"kubernetes.io/projected/0e12a819-fff9-494a-98f3-6df6aba43abe-kube-api-access-qvbd7\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.035808 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-hostroot\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.035808 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-conf-dir\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.035808 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5997abbb-1687-40dc-bdcf-84e52a57673a-agent-certs\") pod \"konnectivity-agent-ggpnx\" (UID: \"5997abbb-1687-40dc-bdcf-84e52a57673a\") " pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:11.035951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:11.035951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-host\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.035951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-var-lib-kubelet\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.036072 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-sys-fs\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.036072 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-var-lib-cni-bin\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.036072 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.035988    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-run-ovn\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.036072 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-host\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.036072 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036037    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-sys-fs\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.036072 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036054    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-var-lib-kubelet\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.036072 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-cni-netd\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-registration-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-registration-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036139    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tqm\" (UniqueName: \"kubernetes.io/projected/d34f692c-575e-4172-9824-2e3852f1e0de-kube-api-access-l7tqm\") pod \"iptables-alerter-6p8sc\" (UID: \"d34f692c-575e-4172-9824-2e3852f1e0de\") " pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-systemd-units\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036174    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-etc-openvswitch\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036195    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-ovnkube-script-lib\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-kubernetes\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036225    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0020ef57-1c96-4818-a96e-6f545c206074-tmp\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpdv\" (UniqueName: \"kubernetes.io/projected/e2f6a821-b6ef-46cd-8fb9-9116f71fa505-kube-api-access-2cpdv\") pod \"node-ca-c4tpn\" (UID: \"e2f6a821-b6ef-46cd-8fb9-9116f71fa505\") " pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.036296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036290    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d34f692c-575e-4172-9824-2e3852f1e0de-host-slash\") pod \"iptables-alerter-6p8sc\" (UID: \"d34f692c-575e-4172-9824-2e3852f1e0de\") " pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4pd4\" (UniqueName: \"kubernetes.io/projected/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-kube-api-access-j4pd4\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036318    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-sysctl-conf\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036332    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-kubernetes\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036357    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46grl\" (UniqueName: \"kubernetes.io/projected/0020ef57-1c96-4818-a96e-6f545c206074-kube-api-access-46grl\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-sysctl-conf\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-cni-dir\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ce228a8-7497-4cb3-9d95-eb279938f9af-cni-binary-copy\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.036488    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.036485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.036767 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.036575    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret podName:e30e502c-ff23-4cad-99f1-5c1b501b3fb4 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:11.536553192 +0000 UTC m=+2.013613126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret") pod "global-pull-secret-syncer-g9xjj" (UID: "e30e502c-ff23-4cad-99f1-5c1b501b3fb4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037005    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-run-multus-certs\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjxd\" (UniqueName: \"kubernetes.io/projected/e47850cc-f2da-4235-8c44-db09fe594725-kube-api-access-xzjxd\") pod \"node-resolver-25clg\" (UID: \"e47850cc-f2da-4235-8c44-db09fe594725\") " pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-sys\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0020ef57-1c96-4818-a96e-6f545c206074-etc-tuned\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfn2\" (UniqueName: \"kubernetes.io/projected/34c67475-95fc-480f-b867-0549df8aff57-kube-api-access-wmfn2\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-sys\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2f6a821-b6ef-46cd-8fb9-9116f71fa505-host\") pod \"node-ca-c4tpn\" (UID: \"e2f6a821-b6ef-46cd-8fb9-9116f71fa505\") " pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037220    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2f6a821-b6ef-46cd-8fb9-9116f71fa505-serviceca\") pod \"node-ca-c4tpn\" (UID: \"e2f6a821-b6ef-46cd-8fb9-9116f71fa505\") " pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-run-openvswitch\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037233    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 14:42:11.037266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-node-log\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rjn\" (UniqueName: \"kubernetes.io/projected/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-kube-api-access-49rjn\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-run\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-run-k8s-cni-cncf-io\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5997abbb-1687-40dc-bdcf-84e52a57673a-konnectivity-ca\") pod \"konnectivity-agent-ggpnx\" (UID: \"5997abbb-1687-40dc-bdcf-84e52a57673a\") " pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-run-netns\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-cni-bin\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-run\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-cnibin\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-var-lib-kubelet\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-modprobe-d\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e12a819-fff9-494a-98f3-6df6aba43abe-system-cni-dir\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e12a819-fff9-494a-98f3-6df6aba43abe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.037768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.037637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-daemon-config\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.038315 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e47850cc-f2da-4235-8c44-db09fe594725-hosts-file\") pod \"node-resolver-25clg\" (UID: \"e47850cc-f2da-4235-8c44-db09fe594725\") " pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.038315 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-kubelet\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.038315 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-lib-modules\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.038315 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038223    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-kubelet-config\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:11.038315 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-socket-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.038315 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038287    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e12a819-fff9-494a-98f3-6df6aba43abe-cni-binary-copy\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.038603 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038319    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e12a819-fff9-494a-98f3-6df6aba43abe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.038603 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038350    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-socket-dir-parent\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.038603 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-slash\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.038603 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-modprobe-d\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.038603 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-socket-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.038603 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-lib-modules\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.038864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-kubelet-config\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:11.038864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038700    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.038864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038720    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47850cc-f2da-4235-8c44-db09fe594725-tmp-dir\") pod \"node-resolver-25clg\" (UID: \"e47850cc-f2da-4235-8c44-db09fe594725\") " pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.038864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-ovn-node-metrics-cert\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.038864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-sysconfig\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.038864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-dbus\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:11.038864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-system-cni-dir\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.038864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-run-netns\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.038864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb74r\" (UniqueName: \"kubernetes.io/projected/5ce228a8-7497-4cb3-9d95-eb279938f9af-kube-api-access-vb74r\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d34f692c-575e-4172-9824-2e3852f1e0de-iptables-alerter-script\") pod \"iptables-alerter-6p8sc\" (UID: \"d34f692c-575e-4172-9824-2e3852f1e0de\") " pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-sysctl-d\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-systemd\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-device-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.038999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e12a819-fff9-494a-98f3-6df6aba43abe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-var-lib-cni-multus\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-run-systemd\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-log-socket\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-env-overrides\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039142    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e12a819-fff9-494a-98f3-6df6aba43abe-cnibin\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-etc-kubernetes\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039203    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-var-lib-openvswitch\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.039260 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-ovnkube-config\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.039848 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-sysconfig\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.039848 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-dbus\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:11.039848 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-sysctl-d\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.039848 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0020ef57-1c96-4818-a96e-6f545c206074-etc-systemd\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.039848 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.039599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-device-dir\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.042075 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.042047    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0020ef57-1c96-4818-a96e-6f545c206074-tmp\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.042188 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.042106    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0020ef57-1c96-4818-a96e-6f545c206074-etc-tuned\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.050016 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.049993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46grl\" (UniqueName: \"kubernetes.io/projected/0020ef57-1c96-4818-a96e-6f545c206074-kube-api-access-46grl\") pod \"tuned-w2rd5\" (UID: \"0020ef57-1c96-4818-a96e-6f545c206074\") " pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.050097 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.050024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rjn\" (UniqueName: \"kubernetes.io/projected/cfe0bfcf-6543-4861-9b3b-bcf642fd0c21-kube-api-access-49rjn\") pod \"aws-ebs-csi-driver-node-2wxfn\" (UID: \"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.052381 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.052361    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:42:11.140418 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-etc-openvswitch\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.140418 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-ovnkube-script-lib\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.140681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpdv\" (UniqueName: \"kubernetes.io/projected/e2f6a821-b6ef-46cd-8fb9-9116f71fa505-kube-api-access-2cpdv\") pod \"node-ca-c4tpn\" (UID: \"e2f6a821-b6ef-46cd-8fb9-9116f71fa505\") " pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.140681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140467    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d34f692c-575e-4172-9824-2e3852f1e0de-host-slash\") pod \"iptables-alerter-6p8sc\" (UID: \"d34f692c-575e-4172-9824-2e3852f1e0de\") " pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.140681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4pd4\" (UniqueName: \"kubernetes.io/projected/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-kube-api-access-j4pd4\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.140681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-etc-openvswitch\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.140681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d34f692c-575e-4172-9824-2e3852f1e0de-host-slash\") pod \"iptables-alerter-6p8sc\" (UID: \"d34f692c-575e-4172-9824-2e3852f1e0de\") " pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.140681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140662    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-cni-dir\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.140949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ce228a8-7497-4cb3-9d95-eb279938f9af-cni-binary-copy\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.140949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-run-multus-certs\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.140949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-run-multus-certs\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.140949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjxd\" (UniqueName: \"kubernetes.io/projected/e47850cc-f2da-4235-8c44-db09fe594725-kube-api-access-xzjxd\") pod \"node-resolver-25clg\" (UID: \"e47850cc-f2da-4235-8c44-db09fe594725\") " pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.140949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-cni-dir\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.140949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfn2\" (UniqueName: \"kubernetes.io/projected/34c67475-95fc-480f-b867-0549df8aff57-kube-api-access-wmfn2\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:11.140949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140899    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2f6a821-b6ef-46cd-8fb9-9116f71fa505-host\") pod \"node-ca-c4tpn\" (UID: \"e2f6a821-b6ef-46cd-8fb9-9116f71fa505\") " pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.140949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2f6a821-b6ef-46cd-8fb9-9116f71fa505-serviceca\") pod \"node-ca-c4tpn\" (UID: \"e2f6a821-b6ef-46cd-8fb9-9116f71fa505\") " pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e2f6a821-b6ef-46cd-8fb9-9116f71fa505-host\") pod \"node-ca-c4tpn\" (UID: \"e2f6a821-b6ef-46cd-8fb9-9116f71fa505\") " pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-run-openvswitch\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.140998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-node-log\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141020    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-run-k8s-cni-cncf-io\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141035    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-run-openvswitch\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5997abbb-1687-40dc-bdcf-84e52a57673a-konnectivity-ca\") pod \"konnectivity-agent-ggpnx\" (UID: \"5997abbb-1687-40dc-bdcf-84e52a57673a\") " pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-run-netns\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-ovnkube-script-lib\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-cni-bin\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-cnibin\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141122    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-var-lib-kubelet\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141121    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-node-log\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-var-lib-kubelet\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-run-netns\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-cnibin\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.141332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e12a819-fff9-494a-98f3-6df6aba43abe-system-cni-dir\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-run-k8s-cni-cncf-io\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-cni-bin\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141214    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e12a819-fff9-494a-98f3-6df6aba43abe-system-cni-dir\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e12a819-fff9-494a-98f3-6df6aba43abe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5ce228a8-7497-4cb3-9d95-eb279938f9af-cni-binary-copy\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-daemon-config\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141323    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e47850cc-f2da-4235-8c44-db09fe594725-hosts-file\") pod \"node-resolver-25clg\" (UID: \"e47850cc-f2da-4235-8c44-db09fe594725\") " pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e2f6a821-b6ef-46cd-8fb9-9116f71fa505-serviceca\") pod \"node-ca-c4tpn\" (UID: \"e2f6a821-b6ef-46cd-8fb9-9116f71fa505\") " pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e12a819-fff9-494a-98f3-6df6aba43abe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141388    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-kubelet\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e47850cc-f2da-4235-8c44-db09fe594725-hosts-file\") pod \"node-resolver-25clg\" (UID: \"e47850cc-f2da-4235-8c44-db09fe594725\") " pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-kubelet\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141456    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e12a819-fff9-494a-98f3-6df6aba43abe-cni-binary-copy\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e12a819-fff9-494a-98f3-6df6aba43abe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-socket-dir-parent\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-slash\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47850cc-f2da-4235-8c44-db09fe594725-tmp-dir\") pod \"node-resolver-25clg\" (UID: \"e47850cc-f2da-4235-8c44-db09fe594725\") " pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141599    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-socket-dir-parent\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141621    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-ovn-node-metrics-cert\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141646    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-run-ovn-kubernetes\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141649    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-slash\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-system-cni-dir\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-run-netns\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141696    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-system-cni-dir\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb74r\" (UniqueName: \"kubernetes.io/projected/5ce228a8-7497-4cb3-9d95-eb279938f9af-kube-api-access-vb74r\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-daemon-config\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-run-netns\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d34f692c-575e-4172-9824-2e3852f1e0de-iptables-alerter-script\") pod \"iptables-alerter-6p8sc\" (UID: \"d34f692c-575e-4172-9824-2e3852f1e0de\") " pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e12a819-fff9-494a-98f3-6df6aba43abe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-var-lib-cni-multus\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141886    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-run-systemd\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141911    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-log-socket\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141942    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-env-overrides\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.142915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141966    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-var-lib-cni-multus\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141976    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e47850cc-f2da-4235-8c44-db09fe594725-tmp-dir\") pod \"node-resolver-25clg\" (UID: \"e47850cc-f2da-4235-8c44-db09fe594725\") " pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.141985    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e12a819-fff9-494a-98f3-6df6aba43abe-cnibin\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-etc-kubernetes\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142039    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e12a819-fff9-494a-98f3-6df6aba43abe-cni-binary-copy\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-etc-kubernetes\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e12a819-fff9-494a-98f3-6df6aba43abe-cnibin\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0e12a819-fff9-494a-98f3-6df6aba43abe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-run-systemd\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-var-lib-openvswitch\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.142206    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142213    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-ovnkube-config\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.142273    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs podName:34c67475-95fc-480f-b867-0549df8aff57 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:11.642255089 +0000 UTC m=+2.119315022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs") pod "network-metrics-daemon-lk75l" (UID: "34c67475-95fc-480f-b867-0549df8aff57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5997abbb-1687-40dc-bdcf-84e52a57673a-konnectivity-ca\") pod \"konnectivity-agent-ggpnx\" (UID: \"5997abbb-1687-40dc-bdcf-84e52a57673a\") " pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142372    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-var-lib-openvswitch\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d34f692c-575e-4172-9824-2e3852f1e0de-iptables-alerter-script\") pod \"iptables-alerter-6p8sc\" (UID: \"d34f692c-575e-4172-9824-2e3852f1e0de\") " pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.143416 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142411    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbd7\" (UniqueName: \"kubernetes.io/projected/0e12a819-fff9-494a-98f3-6df6aba43abe-kube-api-access-qvbd7\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-hostroot\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-conf-dir\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-env-overrides\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5997abbb-1687-40dc-bdcf-84e52a57673a-agent-certs\") pod \"konnectivity-agent-ggpnx\" (UID: \"5997abbb-1687-40dc-bdcf-84e52a57673a\") " pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142486    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0e12a819-fff9-494a-98f3-6df6aba43abe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-hostroot\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142551    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-multus-conf-dir\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-log-socket\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-var-lib-cni-bin\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ce228a8-7497-4cb3-9d95-eb279938f9af-host-var-lib-cni-bin\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-run-ovn\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-cni-netd\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-run-ovn\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142705    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tqm\" (UniqueName: \"kubernetes.io/projected/d34f692c-575e-4172-9824-2e3852f1e0de-kube-api-access-l7tqm\") pod \"iptables-alerter-6p8sc\" (UID: \"d34f692c-575e-4172-9824-2e3852f1e0de\") " pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142729    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-host-cni-netd\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-systemd-units\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.143950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142740    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-ovnkube-config\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.144453 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.142792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-systemd-units\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.144707 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.144690    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-ovn-node-metrics-cert\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.145422 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.145404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5997abbb-1687-40dc-bdcf-84e52a57673a-agent-certs\") pod \"konnectivity-agent-ggpnx\" (UID: \"5997abbb-1687-40dc-bdcf-84e52a57673a\") " pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:11.150122 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.150098    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb74r\" (UniqueName: \"kubernetes.io/projected/5ce228a8-7497-4cb3-9d95-eb279938f9af-kube-api-access-vb74r\") pod \"multus-vzpmc\" (UID: \"5ce228a8-7497-4cb3-9d95-eb279938f9af\") " pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.150279 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.150258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfn2\" (UniqueName: \"kubernetes.io/projected/34c67475-95fc-480f-b867-0549df8aff57-kube-api-access-wmfn2\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:11.150757 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.150727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4pd4\" (UniqueName: \"kubernetes.io/projected/0aa463e7-8b91-46ea-9d0e-c24eea0ff75a-kube-api-access-j4pd4\") pod \"ovnkube-node-kfmvq\" (UID: \"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a\") " pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.151006 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.150989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjxd\" (UniqueName: \"kubernetes.io/projected/e47850cc-f2da-4235-8c44-db09fe594725-kube-api-access-xzjxd\") pod \"node-resolver-25clg\" (UID: \"e47850cc-f2da-4235-8c44-db09fe594725\") " pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.151226 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.151209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpdv\" (UniqueName: \"kubernetes.io/projected/e2f6a821-b6ef-46cd-8fb9-9116f71fa505-kube-api-access-2cpdv\") pod \"node-ca-c4tpn\" (UID: \"e2f6a821-b6ef-46cd-8fb9-9116f71fa505\") " pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.153243 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.153224    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:42:11.153325 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.153246    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:42:11.153325 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.153258    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ztr8k for pod openshift-network-diagnostics/network-check-target-mdn58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:11.153325 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.153318    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k podName:66388537-9e27-479d-9381-8a8116b234ee nodeName:}" failed. No retries permitted until 2026-09-10 14:42:11.653305538 +0000 UTC m=+2.130365459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ztr8k" (UniqueName: "kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k") pod "network-check-target-mdn58" (UID: "66388537-9e27-479d-9381-8a8116b234ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:11.154253 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.154234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbd7\" (UniqueName: \"kubernetes.io/projected/0e12a819-fff9-494a-98f3-6df6aba43abe-kube-api-access-qvbd7\") pod \"multus-additional-cni-plugins-d9dr7\" (UID: \"0e12a819-fff9-494a-98f3-6df6aba43abe\") " pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.155199 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.155180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tqm\" (UniqueName: \"kubernetes.io/projected/d34f692c-575e-4172-9824-2e3852f1e0de-kube-api-access-l7tqm\") pod \"iptables-alerter-6p8sc\" (UID: \"d34f692c-575e-4172-9824-2e3852f1e0de\") " pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.209469 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.209289    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143d3343a56237b77a570303f223bfcd.slice/crio-2318725e31d8848a3d31d969a2bf20abb46c300b700ef8410aff82066de8d55b WatchSource:0}: Error finding container 2318725e31d8848a3d31d969a2bf20abb46c300b700ef8410aff82066de8d55b: Status 404 returned error can't find the container with id 2318725e31d8848a3d31d969a2bf20abb46c300b700ef8410aff82066de8d55b
Sep 10 14:42:11.209702 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.209687    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d04f9a962d39a0595afd41d5bd3b7a.slice/crio-f4ae3ed8b090d115d9b99d82b53ca180438e220af80eb5adccb04d80588ed659 WatchSource:0}: Error finding container f4ae3ed8b090d115d9b99d82b53ca180438e220af80eb5adccb04d80588ed659: Status 404 returned error can't find the container with id f4ae3ed8b090d115d9b99d82b53ca180438e220af80eb5adccb04d80588ed659
Sep 10 14:42:11.213874 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.213856    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:42:11.258460 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.258429    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn"
Sep 10 14:42:11.268659 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.268638    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe0bfcf_6543_4861_9b3b_bcf642fd0c21.slice/crio-51810b62354baefc1bb221d529100e159c1e72bc6d5cb499941eb2b7dc9d4571 WatchSource:0}: Error finding container 51810b62354baefc1bb221d529100e159c1e72bc6d5cb499941eb2b7dc9d4571: Status 404 returned error can't find the container with id 51810b62354baefc1bb221d529100e159c1e72bc6d5cb499941eb2b7dc9d4571
Sep 10 14:42:11.271904 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.271882    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w2rd5"
Sep 10 14:42:11.280083 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.280064    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0020ef57_1c96_4818_a96e_6f545c206074.slice/crio-1d1f3657325062230e3458a0c933c408f6ec47f7346db5d3e7e97efdf45b1d74 WatchSource:0}: Error finding container 1d1f3657325062230e3458a0c933c408f6ec47f7346db5d3e7e97efdf45b1d74: Status 404 returned error can't find the container with id 1d1f3657325062230e3458a0c933c408f6ec47f7346db5d3e7e97efdf45b1d74
Sep 10 14:42:11.280992 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.280972    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d9dr7"
Sep 10 14:42:11.289240 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.289222    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e12a819_fff9_494a_98f3_6df6aba43abe.slice/crio-3edcad4d1e9f8c88e0722a6c112dc58ab853254c00e36721f6a504480af513ac WatchSource:0}: Error finding container 3edcad4d1e9f8c88e0722a6c112dc58ab853254c00e36721f6a504480af513ac: Status 404 returned error can't find the container with id 3edcad4d1e9f8c88e0722a6c112dc58ab853254c00e36721f6a504480af513ac
Sep 10 14:42:11.298173 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.298156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vzpmc"
Sep 10 14:42:11.306776 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.306724    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce228a8_7497_4cb3_9d95_eb279938f9af.slice/crio-e5423b71025ae1c2dfeaf087f08bbc287f3bb46ae42b101731311d6efdc5114e WatchSource:0}: Error finding container e5423b71025ae1c2dfeaf087f08bbc287f3bb46ae42b101731311d6efdc5114e: Status 404 returned error can't find the container with id e5423b71025ae1c2dfeaf087f08bbc287f3bb46ae42b101731311d6efdc5114e
Sep 10 14:42:11.314499 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.314479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:11.321908 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.321886    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5997abbb_1687_40dc_bdcf_84e52a57673a.slice/crio-003b9141b812511f1a2a24592186b661ef3fa3f73145a955ffc368e122a36e43 WatchSource:0}: Error finding container 003b9141b812511f1a2a24592186b661ef3fa3f73145a955ffc368e122a36e43: Status 404 returned error can't find the container with id 003b9141b812511f1a2a24592186b661ef3fa3f73145a955ffc368e122a36e43
Sep 10 14:42:11.328068 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.328053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-25clg"
Sep 10 14:42:11.335651 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.335626    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode47850cc_f2da_4235_8c44_db09fe594725.slice/crio-9eaeb3866195abfb0b03f785c21fb88742edf49709536319e95c495ad7ef464d WatchSource:0}: Error finding container 9eaeb3866195abfb0b03f785c21fb88742edf49709536319e95c495ad7ef464d: Status 404 returned error can't find the container with id 9eaeb3866195abfb0b03f785c21fb88742edf49709536319e95c495ad7ef464d
Sep 10 14:42:11.345795 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.345777    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c4tpn"
Sep 10 14:42:11.354550 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.354531    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f6a821_b6ef_46cd_8fb9_9116f71fa505.slice/crio-a52d10489977f85f93531b7f40789937b6bab867e58546bea8b46450f556a0ae WatchSource:0}: Error finding container a52d10489977f85f93531b7f40789937b6bab867e58546bea8b46450f556a0ae: Status 404 returned error can't find the container with id a52d10489977f85f93531b7f40789937b6bab867e58546bea8b46450f556a0ae
Sep 10 14:42:11.365816 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.365796    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6p8sc"
Sep 10 14:42:11.369397 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.369384    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:11.375276 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.375255    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34f692c_575e_4172_9824_2e3852f1e0de.slice/crio-e12be44ac18c33f0dfc4146e7191e7088c1de7d449614239b5832015b4ba29fc WatchSource:0}: Error finding container e12be44ac18c33f0dfc4146e7191e7088c1de7d449614239b5832015b4ba29fc: Status 404 returned error can't find the container with id e12be44ac18c33f0dfc4146e7191e7088c1de7d449614239b5832015b4ba29fc
Sep 10 14:42:11.377985 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:11.377965    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa463e7_8b91_46ea_9d0e_c24eea0ff75a.slice/crio-c1e19aa1f43931ddccec7cffbb3d3e2974cea35e4c1d49db4b673830ceea6cc7 WatchSource:0}: Error finding container c1e19aa1f43931ddccec7cffbb3d3e2974cea35e4c1d49db4b673830ceea6cc7: Status 404 returned error can't find the container with id c1e19aa1f43931ddccec7cffbb3d3e2974cea35e4c1d49db4b673830ceea6cc7
Sep 10 14:42:11.379959 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.379940    2563 kuberuntime_manager.go:1459] "Unhandled Error" err="container ovn-controller start failed in pod ovnkube-node-kfmvq_openshift-ovn-kubernetes(0aa463e7-8b91-46ea-9d0e-c24eea0ff75a): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 10 14:42:11.381166 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.381149    2563 kuberuntime_manager.go:1459] "Unhandled Error" err="container kube-rbac-proxy-node start failed in pod ovnkube-node-kfmvq_openshift-ovn-kubernetes(0aa463e7-8b91-46ea-9d0e-c24eea0ff75a): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 10 14:42:11.384349 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.384307    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ErrImagePull: \"pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" podUID="0aa463e7-8b91-46ea-9d0e-c24eea0ff75a"
Sep 10 14:42:11.546136 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.546057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:11.546283 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.546153    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:11.546283 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.546197    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret podName:e30e502c-ff23-4cad-99f1-5c1b501b3fb4 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:12.54618598 +0000 UTC m=+3.023245907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret") pod "global-pull-secret-syncer-g9xjj" (UID: "e30e502c-ff23-4cad-99f1-5c1b501b3fb4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:11.646454 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.646423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:11.646650 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.646601    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:11.646717 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.646666    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs podName:34c67475-95fc-480f-b867-0549df8aff57 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:12.6466478 +0000 UTC m=+3.123707729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs") pod "network-metrics-daemon-lk75l" (UID: "34c67475-95fc-480f-b867-0549df8aff57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:11.746703 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.746676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:11.746865 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.746832    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:42:11.746865 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.746852    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:42:11.746865 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.746861    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ztr8k for pod openshift-network-diagnostics/network-check-target-mdn58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:11.746975 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:11.746907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k podName:66388537-9e27-479d-9381-8a8116b234ee nodeName:}" failed. No retries permitted until 2026-09-10 14:42:12.746893449 +0000 UTC m=+3.223953375 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztr8k" (UniqueName: "kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k") pod "network-check-target-mdn58" (UID: "66388537-9e27-479d-9381-8a8116b234ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:11.978940 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.978834    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:37:10 +0000 UTC" deadline="2028-06-13 08:57:08.755089038 +0000 UTC"
Sep 10 14:42:11.978940 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:11.978895    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15402h14m56.776198539s"
Sep 10 14:42:12.035486 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.035390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4tpn" event={"ID":"e2f6a821-b6ef-46cd-8fb9-9116f71fa505","Type":"ContainerStarted","Data":"a52d10489977f85f93531b7f40789937b6bab867e58546bea8b46450f556a0ae"}
Sep 10 14:42:12.037574 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.037490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal" event={"ID":"c8d04f9a962d39a0595afd41d5bd3b7a","Type":"ContainerStarted","Data":"f4ae3ed8b090d115d9b99d82b53ca180438e220af80eb5adccb04d80588ed659"}
Sep 10 14:42:12.039651 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.039590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal" event={"ID":"143d3343a56237b77a570303f223bfcd","Type":"ContainerStarted","Data":"2318725e31d8848a3d31d969a2bf20abb46c300b700ef8410aff82066de8d55b"}
Sep 10 14:42:12.042123 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.041204    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" event={"ID":"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a","Type":"ContainerStarted","Data":"c1e19aa1f43931ddccec7cffbb3d3e2974cea35e4c1d49db4b673830ceea6cc7"}
Sep 10 14:42:12.047278 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.047205    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6p8sc" event={"ID":"d34f692c-575e-4172-9824-2e3852f1e0de","Type":"ContainerStarted","Data":"e12be44ac18c33f0dfc4146e7191e7088c1de7d449614239b5832015b4ba29fc"}
Sep 10 14:42:12.050688 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.050634    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-25clg" event={"ID":"e47850cc-f2da-4235-8c44-db09fe594725","Type":"ContainerStarted","Data":"9eaeb3866195abfb0b03f785c21fb88742edf49709536319e95c495ad7ef464d"}
Sep 10 14:42:12.057657 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:12.057588    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="[failed to \"StartContainer\" for \"ovn-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovn-acl-logging\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-node\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"kube-rbac-proxy-ovn-metrics\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"northd\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"nbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"sbdb\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\", failed to \"StartContainer\" for \"ovnkube-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff86545c65970d36d4ced99fee46fd9a5a7590e379152c8c728eb1f29970dce0\\\": ErrImagePull: pull QPS exceeded\"]" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" podUID="0aa463e7-8b91-46ea-9d0e-c24eea0ff75a"
Sep 10 14:42:12.066485 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.066460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ggpnx" event={"ID":"5997abbb-1687-40dc-bdcf-84e52a57673a","Type":"ContainerStarted","Data":"003b9141b812511f1a2a24592186b661ef3fa3f73145a955ffc368e122a36e43"}
Sep 10 14:42:12.081152 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.081078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vzpmc" event={"ID":"5ce228a8-7497-4cb3-9d95-eb279938f9af","Type":"ContainerStarted","Data":"e5423b71025ae1c2dfeaf087f08bbc287f3bb46ae42b101731311d6efdc5114e"}
Sep 10 14:42:12.089553 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.089524    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" event={"ID":"0e12a819-fff9-494a-98f3-6df6aba43abe","Type":"ContainerStarted","Data":"3edcad4d1e9f8c88e0722a6c112dc58ab853254c00e36721f6a504480af513ac"}
Sep 10 14:42:12.092454 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.092406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w2rd5" event={"ID":"0020ef57-1c96-4818-a96e-6f545c206074","Type":"ContainerStarted","Data":"1d1f3657325062230e3458a0c933c408f6ec47f7346db5d3e7e97efdf45b1d74"}
Sep 10 14:42:12.094748 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.094611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn" event={"ID":"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21","Type":"ContainerStarted","Data":"51810b62354baefc1bb221d529100e159c1e72bc6d5cb499941eb2b7dc9d4571"}
Sep 10 14:42:12.307595 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.307502    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:42:12.553116 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.553079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:12.553430 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:12.553394    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:12.553545 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:12.553481    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret podName:e30e502c-ff23-4cad-99f1-5c1b501b3fb4 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:14.553462865 +0000 UTC m=+5.030522791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret") pod "global-pull-secret-syncer-g9xjj" (UID: "e30e502c-ff23-4cad-99f1-5c1b501b3fb4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:12.654433 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.654347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:12.654959 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:12.654923    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:12.655067 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:12.655013    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs podName:34c67475-95fc-480f-b867-0549df8aff57 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:14.654993172 +0000 UTC m=+5.132053099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs") pod "network-metrics-daemon-lk75l" (UID: "34c67475-95fc-480f-b867-0549df8aff57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:12.681626 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.681596    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:42:12.755397 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.755360    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:12.755602 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:12.755582    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:42:12.755678 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:12.755605    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:42:12.755678 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:12.755618    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ztr8k for pod openshift-network-diagnostics/network-check-target-mdn58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:12.755768 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:12.755692    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k podName:66388537-9e27-479d-9381-8a8116b234ee nodeName:}" failed. No retries permitted until 2026-09-10 14:42:14.755673078 +0000 UTC m=+5.232733016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztr8k" (UniqueName: "kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k") pod "network-check-target-mdn58" (UID: "66388537-9e27-479d-9381-8a8116b234ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:12.979482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.979367    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:37:10 +0000 UTC" deadline="2028-05-03 18:01:11.504313011 +0000 UTC"
Sep 10 14:42:12.979482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:12.979429    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14427h18m58.524888291s"
Sep 10 14:42:13.026119 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:13.026089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:13.026307 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:13.026099    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:13.026307 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:13.026211    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:13.026307 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:13.026229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:13.026469 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:13.026312    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:13.026469 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:13.026388    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:13.904212 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:13.904180    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 14:42:14.572780 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:14.572178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:14.572780 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:14.572327    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:14.572780 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:14.572388    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret podName:e30e502c-ff23-4cad-99f1-5c1b501b3fb4 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:18.572369131 +0000 UTC m=+9.049429069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret") pod "global-pull-secret-syncer-g9xjj" (UID: "e30e502c-ff23-4cad-99f1-5c1b501b3fb4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:14.673212 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:14.673174    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:14.673394 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:14.673339    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:14.673459 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:14.673401    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs podName:34c67475-95fc-480f-b867-0549df8aff57 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:18.673383017 +0000 UTC m=+9.150442943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs") pod "network-metrics-daemon-lk75l" (UID: "34c67475-95fc-480f-b867-0549df8aff57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:14.774544 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:14.774496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:14.774708 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:14.774686    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:42:14.774758 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:14.774710    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:42:14.774758 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:14.774720    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ztr8k for pod openshift-network-diagnostics/network-check-target-mdn58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:14.774843 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:14.774779    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k podName:66388537-9e27-479d-9381-8a8116b234ee nodeName:}" failed. No retries permitted until 2026-09-10 14:42:18.774760277 +0000 UTC m=+9.251820202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztr8k" (UniqueName: "kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k") pod "network-check-target-mdn58" (UID: "66388537-9e27-479d-9381-8a8116b234ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:15.026153 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:15.026120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:15.026316 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:15.026120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:15.026316 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:15.026250    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:15.026457 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:15.026346    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:15.026457 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:15.026125    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:15.026584 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:15.026483    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:17.026475 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:17.026438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:17.026945 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:17.026478    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:17.026945 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:17.026479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:17.026945 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:17.026583    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:17.026945 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:17.026726    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:17.026945 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:17.026826    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:18.604172 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:18.604139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:18.604636 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:18.604278    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:18.604636 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:18.604352    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret podName:e30e502c-ff23-4cad-99f1-5c1b501b3fb4 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:26.604334915 +0000 UTC m=+17.081394842 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret") pod "global-pull-secret-syncer-g9xjj" (UID: "e30e502c-ff23-4cad-99f1-5c1b501b3fb4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:18.704599 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:18.704515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:18.704817 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:18.704680    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:18.704817 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:18.704751    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs podName:34c67475-95fc-480f-b867-0549df8aff57 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:26.704732042 +0000 UTC m=+17.181791982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs") pod "network-metrics-daemon-lk75l" (UID: "34c67475-95fc-480f-b867-0549df8aff57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:18.805365 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:18.805327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:18.805543 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:18.805473    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:42:18.805543 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:18.805490    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:42:18.805543 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:18.805502    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ztr8k for pod openshift-network-diagnostics/network-check-target-mdn58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:18.805694 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:18.805569    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k podName:66388537-9e27-479d-9381-8a8116b234ee nodeName:}" failed. No retries permitted until 2026-09-10 14:42:26.805553591 +0000 UTC m=+17.282613516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztr8k" (UniqueName: "kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k") pod "network-check-target-mdn58" (UID: "66388537-9e27-479d-9381-8a8116b234ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:19.025803 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:19.025688    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:19.025803 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:19.025719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:19.025803 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:19.025754    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:19.026024 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:19.025869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:19.026206 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:19.026170    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:19.026345 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:19.026274    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:21.026294 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.026056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:21.026884 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.026066    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:21.026884 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:21.026415    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:21.026884 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:21.026474    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:21.026884 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.026094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:21.026884 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:21.026620    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:21.132498 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.132456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn" event={"ID":"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21","Type":"ContainerStarted","Data":"301909527e53bb21e987e796675136ba4ac3495a392717a7fb058189f290f597"}
Sep 10 14:42:21.134359 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.134284    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c4tpn" event={"ID":"e2f6a821-b6ef-46cd-8fb9-9116f71fa505","Type":"ContainerStarted","Data":"344e2e08178da874bc4e4d88338dae8dff77126d6d8bb67e5c39b0bbb6f6acaf"}
Sep 10 14:42:21.136058 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.136024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal" event={"ID":"c8d04f9a962d39a0595afd41d5bd3b7a","Type":"ContainerStarted","Data":"d2ebaa353885be726e986980896968479d633853bb06a5810f689e113b1e36bb"}
Sep 10 14:42:21.137913 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.137892    2563 generic.go:356] "Generic (PLEG): container finished" podID="143d3343a56237b77a570303f223bfcd" containerID="5c7635074ed3e79c62008a23637992a85f1eb69b46091cdbb2489f632f8abc89" exitCode=0
Sep 10 14:42:21.138015 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.137955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal" event={"ID":"143d3343a56237b77a570303f223bfcd","Type":"ContainerDied","Data":"5c7635074ed3e79c62008a23637992a85f1eb69b46091cdbb2489f632f8abc89"}
Sep 10 14:42:21.139624 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.139601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-25clg" event={"ID":"e47850cc-f2da-4235-8c44-db09fe594725","Type":"ContainerStarted","Data":"e0f7ad9e6003439379bfbd980fa52b693653a4e86e98dc3f7be6138877fb8ec1"}
Sep 10 14:42:21.141341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.141321    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ggpnx" event={"ID":"5997abbb-1687-40dc-bdcf-84e52a57673a","Type":"ContainerStarted","Data":"1bad9c11d9228bda30d7835ce189482ae55fbd406323aaf5c75c0aa9428658c6"}
Sep 10 14:42:21.143162 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.143138    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e12a819-fff9-494a-98f3-6df6aba43abe" containerID="a8f9a8f14d89eb7bfb4b05d45d8813645f696232426d964b2cff0bcef2138aa1" exitCode=0
Sep 10 14:42:21.143246 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.143213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" event={"ID":"0e12a819-fff9-494a-98f3-6df6aba43abe","Type":"ContainerDied","Data":"a8f9a8f14d89eb7bfb4b05d45d8813645f696232426d964b2cff0bcef2138aa1"}
Sep 10 14:42:21.144789 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.144770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w2rd5" event={"ID":"0020ef57-1c96-4818-a96e-6f545c206074","Type":"ContainerStarted","Data":"8b5667acbd249cbec8c9a33e194b94da895280a12cdd188fb377a821ccbbd0d8"}
Sep 10 14:42:21.148911 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.148861    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c4tpn" podStartSLOduration=1.96122748 podStartE2EDuration="11.148843314s" totalImagesPullingTime="9.187615834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:21.147866514 +0000 UTC m=+11.624926461" watchObservedRunningTime="2026-09-10 14:42:21.148843314 +0000 UTC m=+11.625903259"
Sep 10 14:42:21.160380 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.160339    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ggpnx" podStartSLOduration=1.916330343 podStartE2EDuration="11.160332486s" totalImagesPullingTime="9.244002143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:21.160158917 +0000 UTC m=+11.637218872" watchObservedRunningTime="2026-09-10 14:42:21.160332486 +0000 UTC m=+11.637392430"
Sep 10 14:42:21.175573 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.175537    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-25clg" podStartSLOduration=1.944984286 podStartE2EDuration="11.175524948s" totalImagesPullingTime="9.230540662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:21.175218134 +0000 UTC m=+11.652278079" watchObservedRunningTime="2026-09-10 14:42:21.175524948 +0000 UTC m=+11.652584885"
Sep 10 14:42:21.219187 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.219142    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-186.ec2.internal" podStartSLOduration=11.219123947 podStartE2EDuration="11.219123947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:21.218950451 +0000 UTC m=+11.696010395" watchObservedRunningTime="2026-09-10 14:42:21.219123947 +0000 UTC m=+11.696183891"
Sep 10 14:42:21.975553 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.975518    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:21.976158 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.976135    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:21.989219 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:21.989178    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w2rd5" podStartSLOduration=2.674799442 podStartE2EDuration="11.989162393s" totalImagesPullingTime="9.314362951s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:21.233167631 +0000 UTC m=+11.710227575" watchObservedRunningTime="2026-09-10 14:42:21.989162393 +0000 UTC m=+12.466222336"
Sep 10 14:42:23.026332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:23.026289    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:23.026332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:23.026329    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:23.026919 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:23.026294    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:23.026919 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:23.026444    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:23.026919 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:23.026707    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:23.027145 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:23.027120    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:23.149683 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:23.149642    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6p8sc" event={"ID":"d34f692c-575e-4172-9824-2e3852f1e0de","Type":"ContainerStarted","Data":"5891e28735c2c40947a148374fa9ccd024a81f6d7b459aae58176e51e9c6db2d"}
Sep 10 14:42:23.149842 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:23.149695    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 14:42:25.026379 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.026352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:25.026751 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.026352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:25.026751 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.026354    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:25.026751 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:25.026485    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:25.026751 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:25.026587    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:25.026751 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:25.026666    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:25.157480 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.157437    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vzpmc" event={"ID":"5ce228a8-7497-4cb3-9d95-eb279938f9af","Type":"ContainerStarted","Data":"3cef1a323e25c795fabecee574cd3c5f606e37b6fb740995ea786ba8714b1a09"}
Sep 10 14:42:25.161896 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.161833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal" event={"ID":"143d3343a56237b77a570303f223bfcd","Type":"ContainerStarted","Data":"450401e4d1f053c47f66a819aacd2cf197f40d9cb45c2c247cfcb72e2be04971"}
Sep 10 14:42:25.175895 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.175836    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6p8sc" podStartSLOduration=5.98425832 podStartE2EDuration="15.175816296s" totalImagesPullingTime="9.191557976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:23.163322334 +0000 UTC m=+13.640382278" watchObservedRunningTime="2026-09-10 14:42:25.175816296 +0000 UTC m=+15.652876241"
Sep 10 14:42:25.190804 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.190738    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vzpmc" podStartSLOduration=1.511672144 podStartE2EDuration="15.190715128s" totalImagesPullingTime="13.679042984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:25.17644926 +0000 UTC m=+15.653509207" watchObservedRunningTime="2026-09-10 14:42:25.190715128 +0000 UTC m=+15.667775073"
Sep 10 14:42:25.726202 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.726166    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 14:42:25.851540 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.851425    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:25.851688 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.851648    2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 14:42:25.852092 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.852070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ggpnx"
Sep 10 14:42:25.866997 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.866945    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-186.ec2.internal" podStartSLOduration=15.866926845 podStartE2EDuration="15.866926845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:25.191087552 +0000 UTC m=+15.668147515" watchObservedRunningTime="2026-09-10 14:42:25.866926845 +0000 UTC m=+16.343986790"
Sep 10 14:42:25.993955 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.993810    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T14:42:25.72619464Z","UUID":"b7805435-200a-413c-a2d2-3e1b10acbf9d","Handler":null,"Name":"","Endpoint":""}
Sep 10 14:42:25.996101 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.996075    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 14:42:25.996215 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:25.996119    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 14:42:26.166389 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:26.166307    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn" event={"ID":"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21","Type":"ContainerStarted","Data":"bd83ed76fa42f725baea67113962390a6c7ae570a652726409857dd0034530c4"}
Sep 10 14:42:26.666579 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:26.666544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:26.666850 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:26.666736    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:26.666850 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:26.666802    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret podName:e30e502c-ff23-4cad-99f1-5c1b501b3fb4 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:42.666784035 +0000 UTC m=+33.143843964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret") pod "global-pull-secret-syncer-g9xjj" (UID: "e30e502c-ff23-4cad-99f1-5c1b501b3fb4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:26.767692 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:26.767655    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:26.767881 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:26.767845    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:26.767940 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:26.767907    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs podName:34c67475-95fc-480f-b867-0549df8aff57 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:42.767889525 +0000 UTC m=+33.244949453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs") pod "network-metrics-daemon-lk75l" (UID: "34c67475-95fc-480f-b867-0549df8aff57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:26.868050 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:26.868014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:26.868208 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:26.868189    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:42:26.868279 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:26.868212    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:42:26.868279 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:26.868224    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ztr8k for pod openshift-network-diagnostics/network-check-target-mdn58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:26.868431 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:26.868280    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k podName:66388537-9e27-479d-9381-8a8116b234ee nodeName:}" failed. No retries permitted until 2026-09-10 14:42:42.868260719 +0000 UTC m=+33.345320664 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztr8k" (UniqueName: "kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k") pod "network-check-target-mdn58" (UID: "66388537-9e27-479d-9381-8a8116b234ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:27.026104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:27.026023    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:27.026104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:27.026052    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:27.026331 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:27.026172    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:27.026331 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:27.026235    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:27.026331 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:27.026277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:27.026774 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:27.026359    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:28.173057 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:28.172975    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn" event={"ID":"cfe0bfcf-6543-4861-9b3b-bcf642fd0c21","Type":"ContainerStarted","Data":"6980f5ca35af64cf618a8b9822a3da0e437c96067a4728e6519c960c5f36ee39"}
Sep 10 14:42:28.187583 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:28.187538    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wxfn" podStartSLOduration=2.272617591 podStartE2EDuration="18.187523184s" totalImagesPullingTime="15.914905593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:28.187481293 +0000 UTC m=+18.664541247" watchObservedRunningTime="2026-09-10 14:42:28.187523184 +0000 UTC m=+18.664583121"
Sep 10 14:42:29.025826 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:29.025797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:29.025998 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:29.025797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:29.025998 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:29.025919    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:29.025998 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:29.025797    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:29.025998 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:29.025968    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:29.026183 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:29.026086    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:30.179864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:30.179759    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e12a819-fff9-494a-98f3-6df6aba43abe" containerID="b5b76aeae60cfda6cf3d03fea6478d40831f0bf40000b0b2cb88fc3ddab71e12" exitCode=0
Sep 10 14:42:30.179864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:30.179817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" event={"ID":"0e12a819-fff9-494a-98f3-6df6aba43abe","Type":"ContainerDied","Data":"b5b76aeae60cfda6cf3d03fea6478d40831f0bf40000b0b2cb88fc3ddab71e12"}
Sep 10 14:42:31.026129 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:31.026091    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:31.026129 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:31.026120    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:31.026379 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:31.026233    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:31.026379 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:31.026310    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:31.026379 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:31.026344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:31.026489 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:31.026428    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:33.025994 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:33.025959    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:33.026452 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:33.026075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:33.026452 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:33.026098    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:33.026452 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:33.026114    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:33.026452 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:33.026208    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:33.026452 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:33.026288    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:35.026358 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:35.026331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:35.026831 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:35.026433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:35.026831 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:35.026558    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:35.026831 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:35.026437    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:35.026831 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:35.026434    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:35.026831 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:35.026646    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:35.193997 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:35.193858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" event={"ID":"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a","Type":"ContainerStarted","Data":"acd86c37539a200bd9f3daabbe06530a00fecd60cf333dffebf30d9945b09ced"}
Sep 10 14:42:35.196466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:35.196432    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e12a819-fff9-494a-98f3-6df6aba43abe" containerID="c49eb377a15d5fe8b6b73315840c5d2cb435f9b410716d463986afd673b13ddb" exitCode=0
Sep 10 14:42:35.196599 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:35.196473    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" event={"ID":"0e12a819-fff9-494a-98f3-6df6aba43abe","Type":"ContainerDied","Data":"c49eb377a15d5fe8b6b73315840c5d2cb435f9b410716d463986afd673b13ddb"}
Sep 10 14:42:36.203126 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:36.203091    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" event={"ID":"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a","Type":"ContainerStarted","Data":"865e1811fef3c160dea4e1e40c76eaa72814ef918ad78bc8eb1ca4d474b0973d"}
Sep 10 14:42:36.203448 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:36.203187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" event={"ID":"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a","Type":"ContainerStarted","Data":"fc892d3ea2bb32819e25f657ef1ba4423f8c068f1d3a54750bf5e0ed794372ad"}
Sep 10 14:42:36.203448 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:36.203237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" event={"ID":"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a","Type":"ContainerStarted","Data":"f9315bf2eb9d587fdf6bd3ea069ecfbdccdf6c838d2078fda101a9e623a0f233"}
Sep 10 14:42:36.203448 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:36.203285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" event={"ID":"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a","Type":"ContainerStarted","Data":"ca16882e7f648de757f9a9c7d7940a02a33feeed11e5081560fdddf56f2e8333"}
Sep 10 14:42:36.203448 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:36.203331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" event={"ID":"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a","Type":"ContainerStarted","Data":"38aa60da13075977020f2adfb7c6b01507becf5d9c051c765b7f1cb6861d6e68"}
Sep 10 14:42:37.025826 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:37.025790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:37.026018 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:37.025790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:37.026018 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:37.025919    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:37.026018 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:37.025958    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:37.026169 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:37.026034    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:37.026169 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:37.026095    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:37.207601 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:37.207567    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e12a819-fff9-494a-98f3-6df6aba43abe" containerID="e3314022aebfe813e9c64a7c6344fe275b14906e153649f86d048e463ae86109" exitCode=0
Sep 10 14:42:37.208029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:37.207615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" event={"ID":"0e12a819-fff9-494a-98f3-6df6aba43abe","Type":"ContainerDied","Data":"e3314022aebfe813e9c64a7c6344fe275b14906e153649f86d048e463ae86109"}
Sep 10 14:42:38.214478 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:38.214254    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" event={"ID":"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a","Type":"ContainerStarted","Data":"d97bfdec6d49609450c524c8dd89aacbd7b9f1f2719d6661e8cc549d4347e5e0"}
Sep 10 14:42:39.026492 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:39.026462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:39.026660 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:39.026460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:39.026660 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:39.026460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:39.026660 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:39.026618    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:39.026762 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:39.026704    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:39.026795 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:39.026771    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:40.223941 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:40.223901    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" event={"ID":"0aa463e7-8b91-46ea-9d0e-c24eea0ff75a","Type":"ContainerStarted","Data":"011e57e159bd80c45a328391d50fef50a528d5e0ab01231e69fbe48cf77593d6"}
Sep 10 14:42:40.224755 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:40.224184    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:40.224755 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:40.224245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:40.224755 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:40.224292    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:40.242068 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:40.242016    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:40.242228 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:40.242149    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:42:40.248912 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:40.248867    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq" podStartSLOduration=6.572191534 podStartE2EDuration="30.248848164s" totalImagesPullingTime="23.67665663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=2 observedRunningTime="2026-09-10 14:42:40.248275303 +0000 UTC m=+30.725335273" watchObservedRunningTime="2026-09-10 14:42:40.248848164 +0000 UTC m=+30.725908109"
Sep 10 14:42:41.026575 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:41.026541    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:41.026575 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:41.026564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:41.026773 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:41.026568    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:41.026773 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:41.026669    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:41.026773 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:41.026736    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:41.026903 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:41.026849    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:41.590055 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:41.590019    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lk75l"]
Sep 10 14:42:41.590535 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:41.590187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:41.590535 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:41.590292    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:41.593856 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:41.593829    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdn58"]
Sep 10 14:42:41.594062 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:41.593946    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:41.594143 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:41.594077    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:41.594780 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:41.594449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g9xjj"]
Sep 10 14:42:41.594780 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:41.594558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:41.594780 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:41.594643    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:42.679653 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:42.679617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:42.680084 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:42.679785    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:42.680084 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:42.679848    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret podName:e30e502c-ff23-4cad-99f1-5c1b501b3fb4 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:14.679830705 +0000 UTC m=+65.156890627 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret") pod "global-pull-secret-syncer-g9xjj" (UID: "e30e502c-ff23-4cad-99f1-5c1b501b3fb4") : object "kube-system"/"original-pull-secret" not registered
Sep 10 14:42:42.780582 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:42.780548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:42.780743 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:42.780699    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:42.780792 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:42.780763    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs podName:34c67475-95fc-480f-b867-0549df8aff57 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:14.780744163 +0000 UTC m=+65.257804098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs") pod "network-metrics-daemon-lk75l" (UID: "34c67475-95fc-480f-b867-0549df8aff57") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 14:42:42.881828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:42.881799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:42.882012 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:42.881945    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 14:42:42.882012 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:42.881962    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 14:42:42.882012 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:42.881971    2563 projected.go:196] Error preparing data for projected volume kube-api-access-ztr8k for pod openshift-network-diagnostics/network-check-target-mdn58: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:42.882125 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:42.882016    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k podName:66388537-9e27-479d-9381-8a8116b234ee nodeName:}" failed. No retries permitted until 2026-09-10 14:43:14.882003414 +0000 UTC m=+65.359063341 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ztr8k" (UniqueName: "kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k") pod "network-check-target-mdn58" (UID: "66388537-9e27-479d-9381-8a8116b234ee") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 14:42:43.026579 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:43.026492    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:43.026579 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:43.026527    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:43.026823 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:43.026613    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:43.026823 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:43.026728    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:43.234995 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:43.234805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" event={"ID":"0e12a819-fff9-494a-98f3-6df6aba43abe","Type":"ContainerStarted","Data":"2cd690c9bfd355a1ee65b8cd78d34a8c2a8d0ab4330d8984bb4dcfeb9e49ceff"}
Sep 10 14:42:44.025984 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:44.025909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:44.026302 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:44.026021    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:44.240136 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:44.240107    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e12a819-fff9-494a-98f3-6df6aba43abe" containerID="2cd690c9bfd355a1ee65b8cd78d34a8c2a8d0ab4330d8984bb4dcfeb9e49ceff" exitCode=0
Sep 10 14:42:44.240293 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:44.240159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" event={"ID":"0e12a819-fff9-494a-98f3-6df6aba43abe","Type":"ContainerDied","Data":"2cd690c9bfd355a1ee65b8cd78d34a8c2a8d0ab4330d8984bb4dcfeb9e49ceff"}
Sep 10 14:42:45.025722 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:45.025689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:45.025909 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:45.025812    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:45.025909 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:45.025858    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:45.026027 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:45.025936    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:45.245440 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:45.245407    2563 generic.go:356] "Generic (PLEG): container finished" podID="0e12a819-fff9-494a-98f3-6df6aba43abe" containerID="01cb4864caa91c577dbee1f80f05751c4f28b4b131fa169ccbb002810624ab3b" exitCode=0
Sep 10 14:42:45.245614 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:45.245461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" event={"ID":"0e12a819-fff9-494a-98f3-6df6aba43abe","Type":"ContainerDied","Data":"01cb4864caa91c577dbee1f80f05751c4f28b4b131fa169ccbb002810624ab3b"}
Sep 10 14:42:46.026584 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:46.026554    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:46.026933 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:46.026658    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-g9xjj" podUID="e30e502c-ff23-4cad-99f1-5c1b501b3fb4"
Sep 10 14:42:46.250858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:46.250770    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" event={"ID":"0e12a819-fff9-494a-98f3-6df6aba43abe","Type":"ContainerStarted","Data":"c2de2b3001f2d0e78d523e1b6e24360e77266e819e33695e745828263f236a11"}
Sep 10 14:42:46.273121 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:46.273076    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d9dr7" podStartSLOduration=6.744901929 podStartE2EDuration="36.273060338s" totalImagesPullingTime="29.528158409s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:46.272455345 +0000 UTC m=+36.749515291" watchObservedRunningTime="2026-09-10 14:42:46.273060338 +0000 UTC m=+36.750120282"
Sep 10 14:42:47.025796 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.025765    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:47.025972 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.025772    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:47.025972 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:47.025850    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdn58" podUID="66388537-9e27-479d-9381-8a8116b234ee"
Sep 10 14:42:47.025972 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:47.025932    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lk75l" podUID="34c67475-95fc-480f-b867-0549df8aff57"
Sep 10 14:42:47.800335 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.800306    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-186.ec2.internal" event="NodeReady"
Sep 10 14:42:47.800786 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.800454    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 14:42:47.837099 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.837020    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59ccbd4756-2q8pl"]
Sep 10 14:42:47.858460 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.858432    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"]
Sep 10 14:42:47.859256 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.859232    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:47.863795 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.863766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 14:42:47.863940 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.863792    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7kccf\""
Sep 10 14:42:47.863940 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.863799    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 14:42:47.863940 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.863880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 14:42:47.864271 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.864253    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 14:42:47.864366 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.864263    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 14:42:47.864435 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.864263    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 14:42:47.874381 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.874364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:47.874381 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.874368    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"]
Sep 10 14:42:47.877010 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.876993    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 14:42:47.877242 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.877224    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s7zkb\""
Sep 10 14:42:47.877242 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.877242    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 14:42:47.877798 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.877782    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 10 14:42:47.877884 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.877849    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 10 14:42:47.890491 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.890470    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-z9mrr"]
Sep 10 14:42:47.890605 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.890594    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:47.892579 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.892551    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 14:42:47.892681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.892560    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gb7kw\""
Sep 10 14:42:47.892681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.892612    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 14:42:47.892852 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.892830    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 14:42:47.900739 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.900720    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 14:42:47.904183 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.904164    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz"]
Sep 10 14:42:47.904276 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.904262    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:47.906241 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.906220    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-k4c88\""
Sep 10 14:42:47.906454 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.906442    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 10 14:42:47.906500 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.906479    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 14:42:47.906562 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.906480    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 10 14:42:47.906695 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.906681    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 14:42:47.911916 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.911899    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 10 14:42:47.925144 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.925124    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"]
Sep 10 14:42:47.925247 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.925233    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz"
Sep 10 14:42:47.927216 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.927196    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:42:47.927216 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.927207    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-527mz\""
Sep 10 14:42:47.927341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.927223    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 14:42:47.949050 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.949027    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"]
Sep 10 14:42:47.949144 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.949130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:47.951114 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.951099    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 14:42:47.951210 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.951179    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 14:42:47.951364 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.951345    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bf7bc\""
Sep 10 14:42:47.951445 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.951407    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:42:47.969059 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.969042    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"]
Sep 10 14:42:47.969285 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.969198    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:47.971295 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.971279    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:42:47.971388 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.971311    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 10 14:42:47.971388 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.971315    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 10 14:42:47.971524 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.971382    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 10 14:42:47.971649 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.971632    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-t8l5p\""
Sep 10 14:42:47.987544 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.987523    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8wpzz"]
Sep 10 14:42:47.987626 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.987612    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:47.989591 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.989574    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 14:42:47.989591 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.989585    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:42:47.989729 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.989578    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 14:42:47.989799 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.989775    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-b9bkk\""
Sep 10 14:42:47.989855 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:47.989792    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 14:42:48.002374 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.002356    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"]
Sep 10 14:42:48.002452 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.002383    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59ccbd4756-2q8pl"]
Sep 10 14:42:48.002452 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.002397    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"]
Sep 10 14:42:48.002452 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.002415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"]
Sep 10 14:42:48.002452 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.002440    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz"]
Sep 10 14:42:48.002590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.002458    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"]
Sep 10 14:42:48.002590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.002475    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-z9mrr"]
Sep 10 14:42:48.002590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.002480    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.002590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.002495    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bvc5f"]
Sep 10 14:42:48.004476 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.004453    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 10 14:42:48.004589 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.004478    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 10 14:42:48.004589 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.004496    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 10 14:42:48.004589 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.004525    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 10 14:42:48.004589 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.004526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9qqp4\""
Sep 10 14:42:48.009012 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.008996    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 10 14:42:48.018326 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.018310    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"]
Sep 10 14:42:48.018425 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.018412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-bvc5f"
Sep 10 14:42:48.020700 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.020682    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 14:42:48.020801 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.020715    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 14:42:48.020801 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.020690    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rbszb\""
Sep 10 14:42:48.024156 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024136    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c545044-fe16-41b2-8245-4854e96cc5d3-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.024248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024182    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vjs\" (UniqueName: \"kubernetes.io/projected/f9b16477-9d87-43a7-9f53-2f5621126de6-kube-api-access-r2vjs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.024248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c545044-fe16-41b2-8245-4854e96cc5d3-tmp\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.024354 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-default-certificate\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.024354 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024281    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzqgq\" (UniqueName: \"kubernetes.io/projected/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-kube-api-access-tzqgq\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:48.024354 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:48.024354 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wl5\" (UniqueName: \"kubernetes.io/projected/9da82716-f9d8-443f-bb14-0552cd0c3a7a-kube-api-access-22wl5\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:48.024354 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024338    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-certificates\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.024690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024361    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9da82716-f9d8-443f-bb14-0552cd0c3a7a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:48.024690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024375    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-bound-sa-token\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.024690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024480    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2c545044-fe16-41b2-8245-4854e96cc5d3-snapshots\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.024690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c545044-fe16-41b2-8245-4854e96cc5d3-serving-cert\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.024690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zzph\" (UniqueName: \"kubernetes.io/projected/2c545044-fe16-41b2-8245-4854e96cc5d3-kube-api-access-2zzph\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.024690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-image-registry-private-configuration\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.024690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.024690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:48.024690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024683    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.024983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kmpm\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-kube-api-access-8kmpm\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.024983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.024983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c545044-fe16-41b2-8245-4854e96cc5d3-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.024983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9b2cf5a-81f8-481d-aa65-92277f3c1328-ca-trust-extracted\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.024983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-trusted-ca\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.024983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-stats-auth\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.024983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024872    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-installation-pull-secrets\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.024983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.024897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqhk\" (UniqueName: \"kubernetes.io/projected/052c643e-2396-4b74-893d-3b1c6f0689c1-kube-api-access-hwqhk\") pod \"volume-data-source-validator-695c895bd6-jzbjz\" (UID: \"052c643e-2396-4b74-893d-3b1c6f0689c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz"
Sep 10 14:42:48.032437 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.032420    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"]
Sep 10 14:42:48.032566 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.032551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.034594 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.034573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 14:42:48.034731 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.034652    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 14:42:48.034731 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.034672    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 14:42:48.034855 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.034753    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 14:42:48.054538 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.054523    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6lmdg"]
Sep 10 14:42:48.054619 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.054571    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"
Sep 10 14:42:48.056707 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.056689    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-m9v79\""
Sep 10 14:42:48.056832 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.056812    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 14:42:48.070749 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.070729    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6lmdg"
Sep 10 14:42:48.072784 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.072769    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s4dwj\""
Sep 10 14:42:48.073031 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.073015    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 14:42:48.073122 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.073080    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 14:42:48.073122 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.073084    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 14:42:48.074086 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.074068    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"]
Sep 10 14:42:48.100776 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.100761    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"]
Sep 10 14:42:48.100886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.100785    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mkrsh"]
Sep 10 14:42:48.100886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.100880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.101054 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.101037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:42:48.103126 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.103103    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:42:48.103126 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.103115    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 14:42:48.103301 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.103125    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 14:42:48.103301 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.103132    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 14:42:48.103400 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.103312    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 14:42:48.115442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.115421    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bvc5f"]
Sep 10 14:42:48.115561 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.115479    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8wpzz"]
Sep 10 14:42:48.115561 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.115498    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mkrsh"]
Sep 10 14:42:48.115561 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.115552    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6lmdg"]
Sep 10 14:42:48.115561 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.115570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"]
Sep 10 14:42:48.115764 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.115588    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"]
Sep 10 14:42:48.115764 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.115602    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"]
Sep 10 14:42:48.115764 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.115711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.117558 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.117542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 14:42:48.117558 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.117550    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 14:42:48.117558 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.117557    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mkqg5\""
Sep 10 14:42:48.125230 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9da82716-f9d8-443f-bb14-0552cd0c3a7a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:48.125336 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krr96\" (UniqueName: \"kubernetes.io/projected/e3084961-1d48-497e-b105-b9ad92744ee5-kube-api-access-krr96\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t5w7p\" (UID: \"e3084961-1d48-497e-b105-b9ad92744ee5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.125336 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-image-registry-private-configuration\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.125336 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125320    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:48.125472 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125344    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.125472 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9b2cf5a-81f8-481d-aa65-92277f3c1328-ca-trust-extracted\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.125472 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.125430    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:48.125627 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.125492    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls podName:9da82716-f9d8-443f-bb14-0552cd0c3a7a nodeName:}" failed. No retries permitted until 2026-09-10 14:42:48.625473593 +0000 UTC m=+39.102533525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f4qtm" (UID: "9da82716-f9d8-443f-bb14-0552cd0c3a7a") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:48.125627 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125484    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-trusted-ca\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.125627 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-stats-auth\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.125627 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125583    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqhk\" (UniqueName: \"kubernetes.io/projected/052c643e-2396-4b74-893d-3b1c6f0689c1-kube-api-access-hwqhk\") pod \"volume-data-source-validator-695c895bd6-jzbjz\" (UID: \"052c643e-2396-4b74-893d-3b1c6f0689c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz"
Sep 10 14:42:48.125627 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125611    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/809cecbd-174e-4ca6-9a14-aa49acfb7e67-serving-cert\") pod \"service-ca-operator-859f94c4b-6tf2s\" (UID: \"809cecbd-174e-4ca6-9a14-aa49acfb7e67\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.125828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a2b0239-057f-4348-8709-2406a8d82fae-trusted-ca\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.125828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5951a370-7f0d-4969-a3ab-f622ac8efd0a-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f4d89b65d-b7xtc\" (UID: \"5951a370-7f0d-4969-a3ab-f622ac8efd0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.125828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3084961-1d48-497e-b105-b9ad92744ee5-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t5w7p\" (UID: \"e3084961-1d48-497e-b105-b9ad92744ee5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.125828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vjs\" (UniqueName: \"kubernetes.io/projected/f9b16477-9d87-43a7-9f53-2f5621126de6-kube-api-access-r2vjs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.125828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c545044-fe16-41b2-8245-4854e96cc5d3-tmp\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.125828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125767    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-certificates\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.125828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-default-certificate\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.125828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqgq\" (UniqueName: \"kubernetes.io/projected/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-kube-api-access-tzqgq\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-bound-sa-token\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv55s\" (UniqueName: \"kubernetes.io/projected/5951a370-7f0d-4969-a3ab-f622ac8efd0a-kube-api-access-hv55s\") pod \"klusterlet-addon-workmgr-6f4d89b65d-b7xtc\" (UID: \"5951a370-7f0d-4969-a3ab-f622ac8efd0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125919    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a2b0239-057f-4348-8709-2406a8d82fae-serving-cert\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.125929    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:48.625916429 +0000 UTC m=+39.102976350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : configmap references non-existent config key: service-ca.crt
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.125987    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmpm\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-kube-api-access-8kmpm\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126041    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a2b0239-057f-4348-8709-2406a8d82fae-config\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2c545044-fe16-41b2-8245-4854e96cc5d3-snapshots\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59spc\" (UniqueName: \"kubernetes.io/projected/809cecbd-174e-4ca6-9a14-aa49acfb7e67-kube-api-access-59spc\") pod \"service-ca-operator-859f94c4b-6tf2s\" (UID: \"809cecbd-174e-4ca6-9a14-aa49acfb7e67\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126112    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9b2cf5a-81f8-481d-aa65-92277f3c1328-ca-trust-extracted\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c545044-fe16-41b2-8245-4854e96cc5d3-serving-cert\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126145    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zzph\" (UniqueName: \"kubernetes.io/projected/2c545044-fe16-41b2-8245-4854e96cc5d3-kube-api-access-2zzph\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126183    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c545044-fe16-41b2-8245-4854e96cc5d3-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.126337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49wk\" (UniqueName: \"kubernetes.io/projected/eda9248c-0628-4934-bb1a-c0c4bff16223-kube-api-access-p49wk\") pod \"network-check-source-685456655f-bvc5f\" (UID: \"eda9248c-0628-4934-bb1a-c0c4bff16223\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bvc5f"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9da82716-f9d8-443f-bb14-0552cd0c3a7a-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.126400    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.126412    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64fb7c8dd9-qjmvl: secret "image-registry-tls" not found
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.126466    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls podName:c9b2cf5a-81f8-481d-aa65-92277f3c1328 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:48.62645248 +0000 UTC m=+39.103512411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls") pod "image-registry-64fb7c8dd9-qjmvl" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328") : secret "image-registry-tls" not found
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92lrn\" (UniqueName: \"kubernetes.io/projected/2a2b0239-057f-4348-8709-2406a8d82fae-kube-api-access-92lrn\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.126558    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/809cecbd-174e-4ca6-9a14-aa49acfb7e67-config\") pod \"service-ca-operator-859f94c4b-6tf2s\" (UID: \"809cecbd-174e-4ca6-9a14-aa49acfb7e67\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126629    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5951a370-7f0d-4969-a3ab-f622ac8efd0a-tmp\") pod \"klusterlet-addon-workmgr-6f4d89b65d-b7xtc\" (UID: \"5951a370-7f0d-4969-a3ab-f622ac8efd0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c545044-fe16-41b2-8245-4854e96cc5d3-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3084961-1d48-497e-b105-b9ad92744ee5-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t5w7p\" (UID: \"e3084961-1d48-497e-b105-b9ad92744ee5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gxxl\" (UniqueName: \"kubernetes.io/projected/5a23a194-6046-4fd3-a502-b1eb7c7df60b-kube-api-access-8gxxl\") pod \"managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq\" (UID: \"5a23a194-6046-4fd3-a502-b1eb7c7df60b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-installation-pull-secrets\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126758    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-certificates\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-trusted-ca\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126790    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:48.127096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wl5\" (UniqueName: \"kubernetes.io/projected/9da82716-f9d8-443f-bb14-0552cd0c3a7a-kube-api-access-22wl5\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:48.127901 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.126838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a23a194-6046-4fd3-a502-b1eb7c7df60b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq\" (UID: \"5a23a194-6046-4fd3-a502-b1eb7c7df60b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"
Sep 10 14:42:48.127901 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.126855    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:48.626838535 +0000 UTC m=+39.103898468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : secret "router-metrics-certs-default" not found
Sep 10 14:42:48.127901 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.127012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2c545044-fe16-41b2-8245-4854e96cc5d3-tmp\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.127901 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.127136    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:42:48.127901 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.127179    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls podName:e95b1904-9052-4ab0-bbb1-7ec5cdf770c7 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:48.627166373 +0000 UTC m=+39.104226298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mfp2b" (UID: "e95b1904-9052-4ab0-bbb1-7ec5cdf770c7") : secret "samples-operator-tls" not found
Sep 10 14:42:48.127901 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.127132    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c545044-fe16-41b2-8245-4854e96cc5d3-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.127901 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.127351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2c545044-fe16-41b2-8245-4854e96cc5d3-snapshots\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.127901 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.127482    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c545044-fe16-41b2-8245-4854e96cc5d3-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.130265 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.130189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2c545044-fe16-41b2-8245-4854e96cc5d3-serving-cert\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.130376 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.130286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-installation-pull-secrets\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.130376 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.130309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-default-certificate\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.130376 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.130351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-stats-auth\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.130613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.130386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-image-registry-private-configuration\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.136035 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.136008    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zzph\" (UniqueName: \"kubernetes.io/projected/2c545044-fe16-41b2-8245-4854e96cc5d3-kube-api-access-2zzph\") pod \"insights-operator-76bcd6ccf5-z9mrr\" (UID: \"2c545044-fe16-41b2-8245-4854e96cc5d3\") " pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.136181 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.136158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqhk\" (UniqueName: \"kubernetes.io/projected/052c643e-2396-4b74-893d-3b1c6f0689c1-kube-api-access-hwqhk\") pod \"volume-data-source-validator-695c895bd6-jzbjz\" (UID: \"052c643e-2396-4b74-893d-3b1c6f0689c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz"
Sep 10 14:42:48.137654 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.137605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-bound-sa-token\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.137994 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.137978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqgq\" (UniqueName: \"kubernetes.io/projected/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-kube-api-access-tzqgq\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:48.138223 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.138206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vjs\" (UniqueName: \"kubernetes.io/projected/f9b16477-9d87-43a7-9f53-2f5621126de6-kube-api-access-r2vjs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.138383 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.138362    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wl5\" (UniqueName: \"kubernetes.io/projected/9da82716-f9d8-443f-bb14-0552cd0c3a7a-kube-api-access-22wl5\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:48.138434 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.138386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmpm\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-kube-api-access-8kmpm\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.188854 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.188829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:48.212746 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.212725    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr"
Sep 10 14:42:48.228048 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228002    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/809cecbd-174e-4ca6-9a14-aa49acfb7e67-config\") pod \"service-ca-operator-859f94c4b-6tf2s\" (UID: \"809cecbd-174e-4ca6-9a14-aa49acfb7e67\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.228048 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5951a370-7f0d-4969-a3ab-f622ac8efd0a-tmp\") pod \"klusterlet-addon-workmgr-6f4d89b65d-b7xtc\" (UID: \"5951a370-7f0d-4969-a3ab-f622ac8efd0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.228186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.228186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228066    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.228186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3084961-1d48-497e-b105-b9ad92744ee5-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t5w7p\" (UID: \"e3084961-1d48-497e-b105-b9ad92744ee5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.228186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gxxl\" (UniqueName: \"kubernetes.io/projected/5a23a194-6046-4fd3-a502-b1eb7c7df60b-kube-api-access-8gxxl\") pod \"managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq\" (UID: \"5a23a194-6046-4fd3-a502-b1eb7c7df60b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"
Sep 10 14:42:48.228186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-ca\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.228186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bab8655-a495-49a2-80e3-9a2177570f06-config-volume\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.228482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8q48\" (UniqueName: \"kubernetes.io/projected/f947de55-ffe1-4f80-920c-25f4212290bc-kube-api-access-c8q48\") pod \"ingress-canary-6lmdg\" (UID: \"f947de55-ffe1-4f80-920c-25f4212290bc\") " pod="openshift-ingress-canary/ingress-canary-6lmdg"
Sep 10 14:42:48.228482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f947de55-ffe1-4f80-920c-25f4212290bc-cert\") pod \"ingress-canary-6lmdg\" (UID: \"f947de55-ffe1-4f80-920c-25f4212290bc\") " pod="openshift-ingress-canary/ingress-canary-6lmdg"
Sep 10 14:42:48.228482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228291    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.228482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5cm\" (UniqueName: \"kubernetes.io/projected/621cf180-3ec9-49bd-b66e-e0da3ba2f968-kube-api-access-zw5cm\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.228482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a23a194-6046-4fd3-a502-b1eb7c7df60b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq\" (UID: \"5a23a194-6046-4fd3-a502-b1eb7c7df60b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"
Sep 10 14:42:48.228482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hf65\" (UniqueName: \"kubernetes.io/projected/0bab8655-a495-49a2-80e3-9a2177570f06-kube-api-access-9hf65\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.228482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5951a370-7f0d-4969-a3ab-f622ac8efd0a-tmp\") pod \"klusterlet-addon-workmgr-6f4d89b65d-b7xtc\" (UID: \"5951a370-7f0d-4969-a3ab-f622ac8efd0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.228482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krr96\" (UniqueName: \"kubernetes.io/projected/e3084961-1d48-497e-b105-b9ad92744ee5-kube-api-access-krr96\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t5w7p\" (UID: \"e3084961-1d48-497e-b105-b9ad92744ee5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/621cf180-3ec9-49bd-b66e-e0da3ba2f968-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/809cecbd-174e-4ca6-9a14-aa49acfb7e67-serving-cert\") pod \"service-ca-operator-859f94c4b-6tf2s\" (UID: \"809cecbd-174e-4ca6-9a14-aa49acfb7e67\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228596    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a2b0239-057f-4348-8709-2406a8d82fae-trusted-ca\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5951a370-7f0d-4969-a3ab-f622ac8efd0a-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f4d89b65d-b7xtc\" (UID: \"5951a370-7f0d-4969-a3ab-f622ac8efd0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/809cecbd-174e-4ca6-9a14-aa49acfb7e67-config\") pod \"service-ca-operator-859f94c4b-6tf2s\" (UID: \"809cecbd-174e-4ca6-9a14-aa49acfb7e67\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3084961-1d48-497e-b105-b9ad92744ee5-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t5w7p\" (UID: \"e3084961-1d48-497e-b105-b9ad92744ee5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228721    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv55s\" (UniqueName: \"kubernetes.io/projected/5951a370-7f0d-4969-a3ab-f622ac8efd0a-kube-api-access-hv55s\") pod \"klusterlet-addon-workmgr-6f4d89b65d-b7xtc\" (UID: \"5951a370-7f0d-4969-a3ab-f622ac8efd0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a2b0239-057f-4348-8709-2406a8d82fae-serving-cert\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a2b0239-057f-4348-8709-2406a8d82fae-config\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59spc\" (UniqueName: \"kubernetes.io/projected/809cecbd-174e-4ca6-9a14-aa49acfb7e67-kube-api-access-59spc\") pod \"service-ca-operator-859f94c4b-6tf2s\" (UID: \"809cecbd-174e-4ca6-9a14-aa49acfb7e67\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-hub\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.228894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228893    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49wk\" (UniqueName: \"kubernetes.io/projected/eda9248c-0628-4934-bb1a-c0c4bff16223-kube-api-access-p49wk\") pod \"network-check-source-685456655f-bvc5f\" (UID: \"eda9248c-0628-4934-bb1a-c0c4bff16223\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bvc5f"
Sep 10 14:42:48.229529 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92lrn\" (UniqueName: \"kubernetes.io/projected/2a2b0239-057f-4348-8709-2406a8d82fae-kube-api-access-92lrn\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.229529 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.228954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bab8655-a495-49a2-80e3-9a2177570f06-tmp-dir\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.229529 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.229252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3084961-1d48-497e-b105-b9ad92744ee5-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t5w7p\" (UID: \"e3084961-1d48-497e-b105-b9ad92744ee5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.229682 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.229658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a2b0239-057f-4348-8709-2406a8d82fae-trusted-ca\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.229889 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.229870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a2b0239-057f-4348-8709-2406a8d82fae-config\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.230893 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.230868    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3084961-1d48-497e-b105-b9ad92744ee5-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t5w7p\" (UID: \"e3084961-1d48-497e-b105-b9ad92744ee5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.231241 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.231209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5a23a194-6046-4fd3-a502-b1eb7c7df60b-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq\" (UID: \"5a23a194-6046-4fd3-a502-b1eb7c7df60b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"
Sep 10 14:42:48.231317 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.231209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/809cecbd-174e-4ca6-9a14-aa49acfb7e67-serving-cert\") pod \"service-ca-operator-859f94c4b-6tf2s\" (UID: \"809cecbd-174e-4ca6-9a14-aa49acfb7e67\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.231317 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.231292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5951a370-7f0d-4969-a3ab-f622ac8efd0a-klusterlet-config\") pod \"klusterlet-addon-workmgr-6f4d89b65d-b7xtc\" (UID: \"5951a370-7f0d-4969-a3ab-f622ac8efd0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.231446 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.231430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a2b0239-057f-4348-8709-2406a8d82fae-serving-cert\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.232380 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.232368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz"
Sep 10 14:42:48.236601 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.236554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krr96\" (UniqueName: \"kubernetes.io/projected/e3084961-1d48-497e-b105-b9ad92744ee5-kube-api-access-krr96\") pod \"kube-storage-version-migrator-operator-8d7dd4675-t5w7p\" (UID: \"e3084961-1d48-497e-b105-b9ad92744ee5\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.236857 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.236832    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gxxl\" (UniqueName: \"kubernetes.io/projected/5a23a194-6046-4fd3-a502-b1eb7c7df60b-kube-api-access-8gxxl\") pod \"managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq\" (UID: \"5a23a194-6046-4fd3-a502-b1eb7c7df60b\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"
Sep 10 14:42:48.237819 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.237532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv55s\" (UniqueName: \"kubernetes.io/projected/5951a370-7f0d-4969-a3ab-f622ac8efd0a-kube-api-access-hv55s\") pod \"klusterlet-addon-workmgr-6f4d89b65d-b7xtc\" (UID: \"5951a370-7f0d-4969-a3ab-f622ac8efd0a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.237819 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.237779    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49wk\" (UniqueName: \"kubernetes.io/projected/eda9248c-0628-4934-bb1a-c0c4bff16223-kube-api-access-p49wk\") pod \"network-check-source-685456655f-bvc5f\" (UID: \"eda9248c-0628-4934-bb1a-c0c4bff16223\") " pod="openshift-network-diagnostics/network-check-source-685456655f-bvc5f"
Sep 10 14:42:48.238489 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.238446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59spc\" (UniqueName: \"kubernetes.io/projected/809cecbd-174e-4ca6-9a14-aa49acfb7e67-kube-api-access-59spc\") pod \"service-ca-operator-859f94c4b-6tf2s\" (UID: \"809cecbd-174e-4ca6-9a14-aa49acfb7e67\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.238668 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.238642    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92lrn\" (UniqueName: \"kubernetes.io/projected/2a2b0239-057f-4348-8709-2406a8d82fae-kube-api-access-92lrn\") pod \"console-operator-85b454874b-8wpzz\" (UID: \"2a2b0239-057f-4348-8709-2406a8d82fae\") " pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.269793 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.269763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:48.277230 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.277205    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"
Sep 10 14:42:48.295451 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.295430    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"
Sep 10 14:42:48.311109 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.310840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-bvc5f"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-ca\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bab8655-a495-49a2-80e3-9a2177570f06-config-volume\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8q48\" (UniqueName: \"kubernetes.io/projected/f947de55-ffe1-4f80-920c-25f4212290bc-kube-api-access-c8q48\") pod \"ingress-canary-6lmdg\" (UID: \"f947de55-ffe1-4f80-920c-25f4212290bc\") " pod="openshift-ingress-canary/ingress-canary-6lmdg"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f947de55-ffe1-4f80-920c-25f4212290bc-cert\") pod \"ingress-canary-6lmdg\" (UID: \"f947de55-ffe1-4f80-920c-25f4212290bc\") " pod="openshift-ingress-canary/ingress-canary-6lmdg"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5cm\" (UniqueName: \"kubernetes.io/projected/621cf180-3ec9-49bd-b66e-e0da3ba2f968-kube-api-access-zw5cm\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.332983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hf65\" (UniqueName: \"kubernetes.io/projected/0bab8655-a495-49a2-80e3-9a2177570f06-kube-api-access-9hf65\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.333077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/621cf180-3ec9-49bd-b66e-e0da3ba2f968-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.333188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-hub\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.333233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bab8655-a495-49a2-80e3-9a2177570f06-tmp-dir\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.333595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bab8655-a495-49a2-80e3-9a2177570f06-tmp-dir\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.333722    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.333777    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls podName:0bab8655-a495-49a2-80e3-9a2177570f06 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:48.833758179 +0000 UTC m=+39.310818102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls") pod "dns-default-mkrsh" (UID: "0bab8655-a495-49a2-80e3-9a2177570f06") : secret "dns-default-metrics-tls" not found
Sep 10 14:42:48.335613 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.335130    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/621cf180-3ec9-49bd-b66e-e0da3ba2f968-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.336618 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.335167    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bab8655-a495-49a2-80e3-9a2177570f06-config-volume\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.340576 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.340048    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:42:48.340576 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.340251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-ca\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.340576 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.340527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f947de55-ffe1-4f80-920c-25f4212290bc-cert\") pod \"ingress-canary-6lmdg\" (UID: \"f947de55-ffe1-4f80-920c-25f4212290bc\") " pod="openshift-ingress-canary/ingress-canary-6lmdg"
Sep 10 14:42:48.343738 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.343604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5cm\" (UniqueName: \"kubernetes.io/projected/621cf180-3ec9-49bd-b66e-e0da3ba2f968-kube-api-access-zw5cm\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.343738 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.343685    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8q48\" (UniqueName: \"kubernetes.io/projected/f947de55-ffe1-4f80-920c-25f4212290bc-kube-api-access-c8q48\") pod \"ingress-canary-6lmdg\" (UID: \"f947de55-ffe1-4f80-920c-25f4212290bc\") " pod="openshift-ingress-canary/ingress-canary-6lmdg"
Sep 10 14:42:48.343902 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.343810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hf65\" (UniqueName: \"kubernetes.io/projected/0bab8655-a495-49a2-80e3-9a2177570f06-kube-api-access-9hf65\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.343902 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.343862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-hub\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.344236 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.344215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.345980 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.345936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/621cf180-3ec9-49bd-b66e-e0da3ba2f968-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57885bc544-p4f8l\" (UID: \"621cf180-3ec9-49bd-b66e-e0da3ba2f968\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.371280 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.371034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"
Sep 10 14:42:48.380412 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.379941    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6lmdg"
Sep 10 14:42:48.421637 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.420997    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:42:48.527849 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.527772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-z9mrr"]
Sep 10 14:42:48.531765 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.531740    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c545044_fe16_41b2_8245_4854e96cc5d3.slice/crio-f65fadf81f81081ba2ed9147312d464692b17e8f1e11888410d22b60a19ef29b WatchSource:0}: Error finding container f65fadf81f81081ba2ed9147312d464692b17e8f1e11888410d22b60a19ef29b: Status 404 returned error can't find the container with id f65fadf81f81081ba2ed9147312d464692b17e8f1e11888410d22b60a19ef29b
Sep 10 14:42:48.553659 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.553638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz"]
Sep 10 14:42:48.556895 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.556872    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052c643e_2396_4b74_893d_3b1c6f0689c1.slice/crio-c8bffd2a6b2ab703fdcb8e58b6178e0e7ee3fb7e6b0ba7302357ecfcd9521516 WatchSource:0}: Error finding container c8bffd2a6b2ab703fdcb8e58b6178e0e7ee3fb7e6b0ba7302357ecfcd9521516: Status 404 returned error can't find the container with id c8bffd2a6b2ab703fdcb8e58b6178e0e7ee3fb7e6b0ba7302357ecfcd9521516
Sep 10 14:42:48.581813 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.581790    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s"]
Sep 10 14:42:48.585002 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.584977    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809cecbd_174e_4ca6_9a14_aa49acfb7e67.slice/crio-768d6ef13fecbc5ac999c5971bf9848519593f7d42bff29696b4291a176ba45e WatchSource:0}: Error finding container 768d6ef13fecbc5ac999c5971bf9848519593f7d42bff29696b4291a176ba45e: Status 404 returned error can't find the container with id 768d6ef13fecbc5ac999c5971bf9848519593f7d42bff29696b4291a176ba45e
Sep 10 14:42:48.602274 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.602253    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p"]
Sep 10 14:42:48.604939 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.604914    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3084961_1d48_497e_b105_b9ad92744ee5.slice/crio-5670d207972bdd400d29d5266b336112eb94f9246a5844c4649e8d3d10e0c2f6 WatchSource:0}: Error finding container 5670d207972bdd400d29d5266b336112eb94f9246a5844c4649e8d3d10e0c2f6: Status 404 returned error can't find the container with id 5670d207972bdd400d29d5266b336112eb94f9246a5844c4649e8d3d10e0c2f6
Sep 10 14:42:48.620672 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.620652    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8wpzz"]
Sep 10 14:42:48.623106 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.623079    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2b0239_057f_4348_8709_2406a8d82fae.slice/crio-b33e2d3072dc7fd0a8222d9250195b4f47b601df67b06090be2a01362faf1812 WatchSource:0}: Error finding container b33e2d3072dc7fd0a8222d9250195b4f47b601df67b06090be2a01362faf1812: Status 404 returned error can't find the container with id b33e2d3072dc7fd0a8222d9250195b4f47b601df67b06090be2a01362faf1812
Sep 10 14:42:48.636312 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.636291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:48.636400 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.636321    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.636400 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.636365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:48.636400 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.636394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:48.636565 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636437    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:48.636565 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.636439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:48.636565 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636486    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:49.636462564 +0000 UTC m=+40.113522489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : configmap references non-existent config key: service-ca.crt
Sep 10 14:42:48.636565 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636521    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:42:48.636565 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636530    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:42:48.636565 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636541    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls podName:9da82716-f9d8-443f-bb14-0552cd0c3a7a nodeName:}" failed. No retries permitted until 2026-09-10 14:42:49.636522601 +0000 UTC m=+40.113582530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f4qtm" (UID: "9da82716-f9d8-443f-bb14-0552cd0c3a7a") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:48.636565 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636545    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64fb7c8dd9-qjmvl: secret "image-registry-tls" not found
Sep 10 14:42:48.636565 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636566    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:49.636555155 +0000 UTC m=+40.113615109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : secret "router-metrics-certs-default" not found
Sep 10 14:42:48.636905 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636608    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:42:48.636905 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636627    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls podName:c9b2cf5a-81f8-481d-aa65-92277f3c1328 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:49.636615625 +0000 UTC m=+40.113675562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls") pod "image-registry-64fb7c8dd9-qjmvl" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328") : secret "image-registry-tls" not found
Sep 10 14:42:48.636905 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.636643    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls podName:e95b1904-9052-4ab0-bbb1-7ec5cdf770c7 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:49.6366333 +0000 UTC m=+40.113693248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mfp2b" (UID: "e95b1904-9052-4ab0-bbb1-7ec5cdf770c7") : secret "samples-operator-tls" not found
Sep 10 14:42:48.643199 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.643175    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-bvc5f"]
Sep 10 14:42:48.645595 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.645575    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda9248c_0628_4934_bb1a_c0c4bff16223.slice/crio-e5084d266a4d3a3debbc3c7d3a6cd7efbb8599cdc687d608e91f0b00443ecdd4 WatchSource:0}: Error finding container e5084d266a4d3a3debbc3c7d3a6cd7efbb8599cdc687d608e91f0b00443ecdd4: Status 404 returned error can't find the container with id e5084d266a4d3a3debbc3c7d3a6cd7efbb8599cdc687d608e91f0b00443ecdd4
Sep 10 14:42:48.674978 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.674950    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq"]
Sep 10 14:42:48.677335 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.677313    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a23a194_6046_4fd3_a502_b1eb7c7df60b.slice/crio-a3a6f4d8eaf8a2f85a26636c082e4355228b3efc76c0d287312e3769524bda1a WatchSource:0}: Error finding container a3a6f4d8eaf8a2f85a26636c082e4355228b3efc76c0d287312e3769524bda1a: Status 404 returned error can't find the container with id a3a6f4d8eaf8a2f85a26636c082e4355228b3efc76c0d287312e3769524bda1a
Sep 10 14:42:48.684004 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.683982    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"]
Sep 10 14:42:48.686469 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.686449    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5951a370_7f0d_4969_a3ab_f622ac8efd0a.slice/crio-6c44a2650b4e9cb1b8d2e810f27d5674b05073199ea680338afbed58467683f4 WatchSource:0}: Error finding container 6c44a2650b4e9cb1b8d2e810f27d5674b05073199ea680338afbed58467683f4: Status 404 returned error can't find the container with id 6c44a2650b4e9cb1b8d2e810f27d5674b05073199ea680338afbed58467683f4
Sep 10 14:42:48.699131 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.699112    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6lmdg"]
Sep 10 14:42:48.701755 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.701733    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf947de55_ffe1_4f80_920c_25f4212290bc.slice/crio-9698aed8ea162b17aa2226ce9bc9cd313b74427d1133ce55c8dd436515fd2949 WatchSource:0}: Error finding container 9698aed8ea162b17aa2226ce9bc9cd313b74427d1133ce55c8dd436515fd2949: Status 404 returned error can't find the container with id 9698aed8ea162b17aa2226ce9bc9cd313b74427d1133ce55c8dd436515fd2949
Sep 10 14:42:48.711888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.711866    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"]
Sep 10 14:42:48.714407 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:42:48.714389    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod621cf180_3ec9_49bd_b66e_e0da3ba2f968.slice/crio-b79a7f87a0d78995a817c5b9352086884b111420a11997818065fc8173c05d3a WatchSource:0}: Error finding container b79a7f87a0d78995a817c5b9352086884b111420a11997818065fc8173c05d3a: Status 404 returned error can't find the container with id b79a7f87a0d78995a817c5b9352086884b111420a11997818065fc8173c05d3a
Sep 10 14:42:48.838047 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:48.837968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:48.838688 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.838121    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:42:48.838688 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:48.838178    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls podName:0bab8655-a495-49a2-80e3-9a2177570f06 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:49.838163998 +0000 UTC m=+40.315223924 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls") pod "dns-default-mkrsh" (UID: "0bab8655-a495-49a2-80e3-9a2177570f06") : secret "dns-default-metrics-tls" not found
Sep 10 14:42:49.025775 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.025731    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:42:49.025944 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.025787    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:42:49.028113 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.028091    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:42:49.028304 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.028291    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnf6m\""
Sep 10 14:42:49.028352 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.028306    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c8srj\""
Sep 10 14:42:49.259728 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.259677    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq" event={"ID":"5a23a194-6046-4fd3-a502-b1eb7c7df60b","Type":"ContainerStarted","Data":"a3a6f4d8eaf8a2f85a26636c082e4355228b3efc76c0d287312e3769524bda1a"}
Sep 10 14:42:49.260977 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.260944    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr" event={"ID":"2c545044-fe16-41b2-8245-4854e96cc5d3","Type":"ContainerStarted","Data":"f65fadf81f81081ba2ed9147312d464692b17e8f1e11888410d22b60a19ef29b"}
Sep 10 14:42:49.262263 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.262238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6lmdg" event={"ID":"f947de55-ffe1-4f80-920c-25f4212290bc","Type":"ContainerStarted","Data":"9698aed8ea162b17aa2226ce9bc9cd313b74427d1133ce55c8dd436515fd2949"}
Sep 10 14:42:49.263252 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.263232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc" event={"ID":"5951a370-7f0d-4969-a3ab-f622ac8efd0a","Type":"ContainerStarted","Data":"6c44a2650b4e9cb1b8d2e810f27d5674b05073199ea680338afbed58467683f4"}
Sep 10 14:42:49.264250 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.264225    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8wpzz" event={"ID":"2a2b0239-057f-4348-8709-2406a8d82fae","Type":"ContainerStarted","Data":"b33e2d3072dc7fd0a8222d9250195b4f47b601df67b06090be2a01362faf1812"}
Sep 10 14:42:49.265276 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.265250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" event={"ID":"621cf180-3ec9-49bd-b66e-e0da3ba2f968","Type":"ContainerStarted","Data":"b79a7f87a0d78995a817c5b9352086884b111420a11997818065fc8173c05d3a"}
Sep 10 14:42:49.266279 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.266262    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p" event={"ID":"e3084961-1d48-497e-b105-b9ad92744ee5","Type":"ContainerStarted","Data":"5670d207972bdd400d29d5266b336112eb94f9246a5844c4649e8d3d10e0c2f6"}
Sep 10 14:42:49.267295 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.267266    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s" event={"ID":"809cecbd-174e-4ca6-9a14-aa49acfb7e67","Type":"ContainerStarted","Data":"768d6ef13fecbc5ac999c5971bf9848519593f7d42bff29696b4291a176ba45e"}
Sep 10 14:42:49.270705 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.270686    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-bvc5f" event={"ID":"eda9248c-0628-4934-bb1a-c0c4bff16223","Type":"ContainerStarted","Data":"e5084d266a4d3a3debbc3c7d3a6cd7efbb8599cdc687d608e91f0b00443ecdd4"}
Sep 10 14:42:49.271599 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.271579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz" event={"ID":"052c643e-2396-4b74-893d-3b1c6f0689c1","Type":"ContainerStarted","Data":"c8bffd2a6b2ab703fdcb8e58b6178e0e7ee3fb7e6b0ba7302357ecfcd9521516"}
Sep 10 14:42:49.645241 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.645203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:49.645410 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.645262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:49.645410 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.645341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:49.645410 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.645392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:49.645624 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.645422    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:49.645675 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.645629    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:51.645610171 +0000 UTC m=+42.122670094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : configmap references non-existent config key: service-ca.crt
Sep 10 14:42:49.645744 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.645730    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:42:49.645793 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.645769    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:51.645757161 +0000 UTC m=+42.122817088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : secret "router-metrics-certs-default" not found
Sep 10 14:42:49.645853 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.645831    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:42:49.645853 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.645841    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64fb7c8dd9-qjmvl: secret "image-registry-tls" not found
Sep 10 14:42:49.645969 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.645870    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls podName:c9b2cf5a-81f8-481d-aa65-92277f3c1328 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:51.645860815 +0000 UTC m=+42.122920743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls") pod "image-registry-64fb7c8dd9-qjmvl" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328") : secret "image-registry-tls" not found
Sep 10 14:42:49.645969 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.645940    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:42:49.645969 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.645969    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls podName:e95b1904-9052-4ab0-bbb1-7ec5cdf770c7 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:51.645960066 +0000 UTC m=+42.123019990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mfp2b" (UID: "e95b1904-9052-4ab0-bbb1-7ec5cdf770c7") : secret "samples-operator-tls" not found
Sep 10 14:42:49.646132 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.646029    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:49.646132 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.646058    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls podName:9da82716-f9d8-443f-bb14-0552cd0c3a7a nodeName:}" failed. No retries permitted until 2026-09-10 14:42:51.646048848 +0000 UTC m=+42.123108774 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f4qtm" (UID: "9da82716-f9d8-443f-bb14-0552cd0c3a7a") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:49.847673 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:49.847636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:49.848137 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.847914    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:42:49.848137 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:49.847979    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls podName:0bab8655-a495-49a2-80e3-9a2177570f06 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:51.847960377 +0000 UTC m=+42.325020322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls") pod "dns-default-mkrsh" (UID: "0bab8655-a495-49a2-80e3-9a2177570f06") : secret "dns-default-metrics-tls" not found
Sep 10 14:42:51.670272 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:51.670040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:51.670678 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:51.670299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:51.670678 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:51.670379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:51.670678 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:51.670446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:51.670678 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:51.670481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:51.670894 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.670700    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:55.670680098 +0000 UTC m=+46.147740038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : configmap references non-existent config key: service-ca.crt
Sep 10 14:42:51.671165 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.671144    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:42:51.671259 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.671203    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:55.671184557 +0000 UTC m=+46.148244493 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : secret "router-metrics-certs-default" not found
Sep 10 14:42:51.671321 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.671266    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:42:51.671321 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.671276    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64fb7c8dd9-qjmvl: secret "image-registry-tls" not found
Sep 10 14:42:51.671321 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.671310    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls podName:c9b2cf5a-81f8-481d-aa65-92277f3c1328 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:55.671299067 +0000 UTC m=+46.148358993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls") pod "image-registry-64fb7c8dd9-qjmvl" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328") : secret "image-registry-tls" not found
Sep 10 14:42:51.671468 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.671382    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:42:51.671468 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.671424    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls podName:e95b1904-9052-4ab0-bbb1-7ec5cdf770c7 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:55.671413563 +0000 UTC m=+46.148473492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mfp2b" (UID: "e95b1904-9052-4ab0-bbb1-7ec5cdf770c7") : secret "samples-operator-tls" not found
Sep 10 14:42:51.671608 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.671479    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:51.671608 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.671529    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls podName:9da82716-f9d8-443f-bb14-0552cd0c3a7a nodeName:}" failed. No retries permitted until 2026-09-10 14:42:55.67150041 +0000 UTC m=+46.148560335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f4qtm" (UID: "9da82716-f9d8-443f-bb14-0552cd0c3a7a") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:51.872660 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:51.872622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:51.873207 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.872923    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:42:51.873207 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:51.872986    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls podName:0bab8655-a495-49a2-80e3-9a2177570f06 nodeName:}" failed. No retries permitted until 2026-09-10 14:42:55.872968647 +0000 UTC m=+46.350028577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls") pod "dns-default-mkrsh" (UID: "0bab8655-a495-49a2-80e3-9a2177570f06") : secret "dns-default-metrics-tls" not found
Sep 10 14:42:53.369061 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:53.368862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:53.467596 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:53.467392    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:54.754525 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:54.754408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:54.852165 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:54.852127    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:55.721882 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:55.721823    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:42:55.722120 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:55.721917    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:42:55.722120 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:55.721953    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:55.722120 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.721980    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:42:55.722120 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:55.722027    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:42:55.722120 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.722057    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls podName:e95b1904-9052-4ab0-bbb1-7ec5cdf770c7 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:03.722040718 +0000 UTC m=+54.199100639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mfp2b" (UID: "e95b1904-9052-4ab0-bbb1-7ec5cdf770c7") : secret "samples-operator-tls" not found
Sep 10 14:42:55.722120 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.722073    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:55.722120 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:55.722097    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:42:55.722120 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.722109    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:42:55.722527 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.722132    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls podName:9da82716-f9d8-443f-bb14-0552cd0c3a7a nodeName:}" failed. No retries permitted until 2026-09-10 14:43:03.722115849 +0000 UTC m=+54.199175772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f4qtm" (UID: "9da82716-f9d8-443f-bb14-0552cd0c3a7a") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:42:55.722527 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.722154    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:03.722145535 +0000 UTC m=+54.199205459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : secret "router-metrics-certs-default" not found
Sep 10 14:42:55.722527 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.722169    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:42:55.722527 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.722181    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64fb7c8dd9-qjmvl: secret "image-registry-tls" not found
Sep 10 14:42:55.722527 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.722170    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:03.722162879 +0000 UTC m=+54.199222801 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : configmap references non-existent config key: service-ca.crt
Sep 10 14:42:55.722527 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.722229    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls podName:c9b2cf5a-81f8-481d-aa65-92277f3c1328 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:03.72221728 +0000 UTC m=+54.199277205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls") pod "image-registry-64fb7c8dd9-qjmvl" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328") : secret "image-registry-tls" not found
Sep 10 14:42:55.924905 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:55.924869    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:42:55.925297 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.924988    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:42:55.925297 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:42:55.925051    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls podName:0bab8655-a495-49a2-80e3-9a2177570f06 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:03.925030324 +0000 UTC m=+54.402090246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls") pod "dns-default-mkrsh" (UID: "0bab8655-a495-49a2-80e3-9a2177570f06") : secret "dns-default-metrics-tls" not found
Sep 10 14:42:58.352422 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:58.352376    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:58.447965 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:58.447929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:58.503187 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:58.503151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:59.363642 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:59.363607    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:42:59.464326 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:42:59.464294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:01.270962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:01.270885    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:01.338441 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:01.338408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:01.961355 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:01.961327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:02.062960 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.062879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:02.322640 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.322548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-bvc5f" event={"ID":"eda9248c-0628-4934-bb1a-c0c4bff16223","Type":"ContainerStarted","Data":"4c7741b0407af3337c5522f34b600815531397a3123d8ac40bca74a0ea4f466a"}
Sep 10 14:43:02.324319 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.324285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz" event={"ID":"052c643e-2396-4b74-893d-3b1c6f0689c1","Type":"ContainerStarted","Data":"b6ad1234532995e969a9dcbbc296b77c29dc0a82b886dc76218bc38f2f4346c9"}
Sep 10 14:43:02.326644 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.326131    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq" event={"ID":"5a23a194-6046-4fd3-a502-b1eb7c7df60b","Type":"ContainerStarted","Data":"1e00b09a394b5d88ea47f82bf2d92c89ff18b40ebef182af30dba2e50afe482b"}
Sep 10 14:43:02.327910 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.327887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr" event={"ID":"2c545044-fe16-41b2-8245-4854e96cc5d3","Type":"ContainerStarted","Data":"259ecf5c5e99bfb9202dd5528273349b11465ff3b7baa18a4de7854e90b0cb35"}
Sep 10 14:43:02.329809 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.329619    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6lmdg" event={"ID":"f947de55-ffe1-4f80-920c-25f4212290bc","Type":"ContainerStarted","Data":"efd68b6ce01b4c881b2e9532774bd9981a0a61a39ac8396f14349349d6b76273"}
Sep 10 14:43:02.331241 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.331213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc" event={"ID":"5951a370-7f0d-4969-a3ab-f622ac8efd0a","Type":"ContainerStarted","Data":"c1eecfa029123108035620f25f5dc3e022efb73af50f127f0a07f162cbe8ee7c"}
Sep 10 14:43:02.331478 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.331462    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:43:02.332715 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.332695    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc"
Sep 10 14:43:02.333770 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.333751    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/0.log"
Sep 10 14:43:02.333923 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.333788    2563 generic.go:356] "Generic (PLEG): container finished" podID="2a2b0239-057f-4348-8709-2406a8d82fae" containerID="e0c5214630417be07c0a2da896f63cc7ea54f2e138be0b4aff9273795f8e3b6d" exitCode=255
Sep 10 14:43:02.333923 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.333866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8wpzz" event={"ID":"2a2b0239-057f-4348-8709-2406a8d82fae","Type":"ContainerDied","Data":"e0c5214630417be07c0a2da896f63cc7ea54f2e138be0b4aff9273795f8e3b6d"}
Sep 10 14:43:02.334028 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.334009    2563 scope.go:117] "RemoveContainer" containerID="e0c5214630417be07c0a2da896f63cc7ea54f2e138be0b4aff9273795f8e3b6d"
Sep 10 14:43:02.336389 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.336361    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" event={"ID":"621cf180-3ec9-49bd-b66e-e0da3ba2f968","Type":"ContainerStarted","Data":"949a92e70fcfd0d37b1ea65dde85ae926ecdfdb76bb7263c4d6e9f0f41740651"}
Sep 10 14:43:02.336493 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.336402    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" event={"ID":"621cf180-3ec9-49bd-b66e-e0da3ba2f968","Type":"ContainerStarted","Data":"7208fcae58b19bb36c406099ccfa86c092f2dc2737d2fde0f6036be82670f1bc"}
Sep 10 14:43:02.336493 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.336445    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" event={"ID":"621cf180-3ec9-49bd-b66e-e0da3ba2f968","Type":"ContainerStarted","Data":"a693bd4e1a8045caf41b97daa8c9b6c784c71105fbff14ce6df1d9502b412272"}
Sep 10 14:43:02.339180 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.338138    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-bvc5f" podStartSLOduration=13.482257229 podStartE2EDuration="26.33812336s" totalImagesPullingTime="12.855866131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:02.337459689 +0000 UTC m=+52.814519635" watchObservedRunningTime="2026-09-10 14:43:02.33812336 +0000 UTC m=+52.815183307"
Sep 10 14:43:02.339760 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.339738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p" event={"ID":"e3084961-1d48-497e-b105-b9ad92744ee5","Type":"ContainerStarted","Data":"ef78a1ff23014ef8fa0cd9f972fc94dcd8d2a584e518fc9734732e0194475439"}
Sep 10 14:43:02.341971 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.341951    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s" event={"ID":"809cecbd-174e-4ca6-9a14-aa49acfb7e67","Type":"ContainerStarted","Data":"5eb9f20eda172a0bbc8f74d9c37c3fc7b139b5e93a3a86e5626e8ea00dc2ea31"}
Sep 10 14:43:02.370304 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.370242    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6f4d89b65d-b7xtc" podStartSLOduration=3.555955716 podStartE2EDuration="16.370220467s" totalImagesPullingTime="12.814264751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:02.351954048 +0000 UTC m=+52.829013993" watchObservedRunningTime="2026-09-10 14:43:02.370220467 +0000 UTC m=+52.847280402"
Sep 10 14:43:02.371435 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.371398    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-jzbjz" podStartSLOduration=27.57589296 podStartE2EDuration="40.371388225s" totalImagesPullingTime="12.795495265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:02.370293431 +0000 UTC m=+52.847353374" watchObservedRunningTime="2026-09-10 14:43:02.371388225 +0000 UTC m=+52.848448173"
Sep 10 14:43:02.409239 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.409193    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" podStartSLOduration=3.62317071 podStartE2EDuration="16.409177658s" totalImagesPullingTime="12.786006948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:02.407955607 +0000 UTC m=+52.885015578" watchObservedRunningTime="2026-09-10 14:43:02.409177658 +0000 UTC m=+52.886237603"
Sep 10 14:43:02.430954 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.430898    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-64c85c4b8d-nj7zq" podStartSLOduration=3.599298038 podStartE2EDuration="16.430878746s" totalImagesPullingTime="12.831580708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:02.429176241 +0000 UTC m=+52.906236187" watchObservedRunningTime="2026-09-10 14:43:02.430878746 +0000 UTC m=+52.907938695"
Sep 10 14:43:02.447123 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.447067    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr" podStartSLOduration=37.476469566 podStartE2EDuration="50.447046798s" totalImagesPullingTime="12.970577232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:02.446031241 +0000 UTC m=+52.923091186" watchObservedRunningTime="2026-09-10 14:43:02.447046798 +0000 UTC m=+52.924106743"
Sep 10 14:43:02.459365 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.459284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:02.479920 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.479040    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6lmdg" podStartSLOduration=2.680220961 podStartE2EDuration="15.479017348s" totalImagesPullingTime="12.798796387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:02.462101359 +0000 UTC m=+52.939161300" watchObservedRunningTime="2026-09-10 14:43:02.479017348 +0000 UTC m=+52.956077293"
Sep 10 14:43:02.480184 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.480145    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p" podStartSLOduration=27.584554962 podStartE2EDuration="40.480130351s" totalImagesPullingTime="12.895575389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:02.47978474 +0000 UTC m=+52.956844697" watchObservedRunningTime="2026-09-10 14:43:02.480130351 +0000 UTC m=+52.957190296"
Sep 10 14:43:02.495499 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.495449    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s" podStartSLOduration=27.581844407 podStartE2EDuration="40.495428148s" totalImagesPullingTime="12.913583741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:02.494637044 +0000 UTC m=+52.971696989" watchObservedRunningTime="2026-09-10 14:43:02.495428148 +0000 UTC m=+52.972488086"
Sep 10 14:43:02.530972 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:02.530940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.060529 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.060476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.065445 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.064263    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5249m"]
Sep 10 14:43:03.067910 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.067501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.069601 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.069579    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 14:43:03.069601 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.069598    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fmx4m\""
Sep 10 14:43:03.069876 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.069857    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 14:43:03.078133 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.078112    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5249m"]
Sep 10 14:43:03.134703 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.132560    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.203883 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.203846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.204047 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.203912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f03b9612-34b5-4495-b105-777d664603fa-data-volume\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.204047 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.203948    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twr5b\" (UniqueName: \"kubernetes.io/projected/f03b9612-34b5-4495-b105-777d664603fa-kube-api-access-twr5b\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.204047 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.203969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.204173 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.204052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f03b9612-34b5-4495-b105-777d664603fa-crio-socket\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.305359 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.305325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f03b9612-34b5-4495-b105-777d664603fa-data-volume\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.305541 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.305375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twr5b\" (UniqueName: \"kubernetes.io/projected/f03b9612-34b5-4495-b105-777d664603fa-kube-api-access-twr5b\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.305541 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.305402    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.305541 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.305488    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f03b9612-34b5-4495-b105-777d664603fa-crio-socket\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.305653 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.305563    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:43:03.305653 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.305574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f03b9612-34b5-4495-b105-777d664603fa-crio-socket\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.305653 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.305646    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:03.805625542 +0000 UTC m=+54.282685479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : configmap "kube-rbac-proxy" not found
Sep 10 14:43:03.305775 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.305746    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.305907 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.305880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f03b9612-34b5-4495-b105-777d664603fa-data-volume\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.305973 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.305896    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:03.306050 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.306035    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:03.806014087 +0000 UTC m=+54.283074014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:03.320198 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.320138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twr5b\" (UniqueName: \"kubernetes.io/projected/f03b9612-34b5-4495-b105-777d664603fa-kube-api-access-twr5b\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.347310 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.347282    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 14:43:03.347837 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.347818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/0.log"
Sep 10 14:43:03.347894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.347859    2563 generic.go:356] "Generic (PLEG): container finished" podID="2a2b0239-057f-4348-8709-2406a8d82fae" containerID="556420248e3e360c56055e29d09f359d0820b5642e13edeee54da9f29a610fee" exitCode=255
Sep 10 14:43:03.348069 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.347977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8wpzz" event={"ID":"2a2b0239-057f-4348-8709-2406a8d82fae","Type":"ContainerDied","Data":"556420248e3e360c56055e29d09f359d0820b5642e13edeee54da9f29a610fee"}
Sep 10 14:43:03.348167 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.348075    2563 scope.go:117] "RemoveContainer" containerID="e0c5214630417be07c0a2da896f63cc7ea54f2e138be0b4aff9273795f8e3b6d"
Sep 10 14:43:03.348276 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.348258    2563 scope.go:117] "RemoveContainer" containerID="556420248e3e360c56055e29d09f359d0820b5642e13edeee54da9f29a610fee"
Sep 10 14:43:03.348550 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.348497    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8wpzz_openshift-console-operator(2a2b0239-057f-4348-8709-2406a8d82fae)\"" pod="openshift-console-operator/console-operator-85b454874b-8wpzz" podUID="2a2b0239-057f-4348-8709-2406a8d82fae"
Sep 10 14:43:03.355563 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.355535    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.446607 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.446573    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.526876 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.526848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.603363 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.603283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.668679 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.668646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.743989 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.743957    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.811620 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.811589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:43:03.811777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.811666    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.811777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.811696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:43:03.811777 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811738    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 14:43:03.811777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.811743    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:43:03.811777 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811753    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64fb7c8dd9-qjmvl: secret "image-registry-tls" not found
Sep 10 14:43:03.811777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.811771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811779    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811809    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls podName:c9b2cf5a-81f8-481d-aa65-92277f3c1328 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:19.811792217 +0000 UTC m=+70.288852139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls") pod "image-registry-64fb7c8dd9-qjmvl" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328") : secret "image-registry-tls" not found
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811823    2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811845    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:04.811828715 +0000 UTC m=+55.288888637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : configmap "kube-rbac-proxy" not found
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811883    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811890    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls podName:e95b1904-9052-4ab0-bbb1-7ec5cdf770c7 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:19.811873066 +0000 UTC m=+70.288932991 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-mfp2b" (UID: "e95b1904-9052-4ab0-bbb1-7ec5cdf770c7") : secret "samples-operator-tls" not found
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811913    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:19.811903849 +0000 UTC m=+70.288963771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : configmap references non-existent config key: service-ca.crt
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.811945    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls podName:9da82716-f9d8-443f-bb14-0552cd0c3a7a nodeName:}" failed. No retries permitted until 2026-09-10 14:43:19.811936292 +0000 UTC m=+70.288996219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-f4qtm" (UID: "9da82716-f9d8-443f-bb14-0552cd0c3a7a") : secret "cluster-monitoring-operator-tls" not found
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.811964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.811996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:03.812095 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.812092    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:03.812479 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.812131    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:04.812120168 +0000 UTC m=+55.289180103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:03.812479 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.812091    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 14:43:03.812479 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:03.812169    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs podName:f9b16477-9d87-43a7-9f53-2f5621126de6 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:19.812160014 +0000 UTC m=+70.289219969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs") pod "router-default-59ccbd4756-2q8pl" (UID: "f9b16477-9d87-43a7-9f53-2f5621126de6") : secret "router-metrics-certs-default" not found
Sep 10 14:43:03.843973 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.843950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:03.945175 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:03.945148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:04.013841 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:04.013815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:43:04.013995 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:04.013930    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 14:43:04.013995 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:04.013977    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls podName:0bab8655-a495-49a2-80e3-9a2177570f06 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:20.013965572 +0000 UTC m=+70.491025494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls") pod "dns-default-mkrsh" (UID: "0bab8655-a495-49a2-80e3-9a2177570f06") : secret "dns-default-metrics-tls" not found
Sep 10 14:43:04.175626 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:04.175598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:04.352327 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:04.352248    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 14:43:04.352802 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:04.352681    2563 scope.go:117] "RemoveContainer" containerID="556420248e3e360c56055e29d09f359d0820b5642e13edeee54da9f29a610fee"
Sep 10 14:43:04.352894 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:04.352872    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8wpzz_openshift-console-operator(2a2b0239-057f-4348-8709-2406a8d82fae)\"" pod="openshift-console-operator/console-operator-85b454874b-8wpzz" podUID="2a2b0239-057f-4348-8709-2406a8d82fae"
Sep 10 14:43:04.407085 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:04.407063    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-25clg_e47850cc-f2da-4235-8c44-db09fe594725/dns-node-resolver/0.log"
Sep 10 14:43:04.821383 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:04.821352    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:04.821553 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:04.821460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:04.821553 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:04.821489    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:04.821681 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:04.821564    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:06.821548481 +0000 UTC m=+57.298608406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:04.821681 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:04.821604    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:43:04.821681 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:04.821670    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:06.821650612 +0000 UTC m=+57.298710550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : configmap "kube-rbac-proxy" not found
Sep 10 14:43:05.354904 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:05.354870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:05.606166 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:05.606095    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4tpn_e2f6a821-b6ef-46cd-8fb9-9116f71fa505/node-ca/0.log"
Sep 10 14:43:06.213303 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:06.213271    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:06.407215 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:06.407184    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6lmdg_f947de55-ffe1-4f80-920c-25f4212290bc/serve-healthcheck-canary/0.log"
Sep 10 14:43:06.840497 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:06.840459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:06.840671 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:06.840569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:06.840671 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:06.840630    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:06.840671 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:06.840653    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:43:06.840765 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:06.840699    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:10.84068308 +0000 UTC m=+61.317743019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:06.840765 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:06.840715    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:10.840708253 +0000 UTC m=+61.317768175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : configmap "kube-rbac-proxy" not found
Sep 10 14:43:08.311562 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:08.311525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:43:08.311930 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:08.311584    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:43:08.311930 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:08.311903    2563 scope.go:117] "RemoveContainer" containerID="556420248e3e360c56055e29d09f359d0820b5642e13edeee54da9f29a610fee"
Sep 10 14:43:08.312107 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:08.312082    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8wpzz_openshift-console-operator(2a2b0239-057f-4348-8709-2406a8d82fae)\"" pod="openshift-console-operator/console-operator-85b454874b-8wpzz" podUID="2a2b0239-057f-4348-8709-2406a8d82fae"
Sep 10 14:43:08.680906 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:08.680874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:09.843100 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:09.843068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:09.961109 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:09.961045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:10.062025 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:10.061990    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:10.162861 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:10.162831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:10.265442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:10.265406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:10.368563 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:10.368501    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:10.743163 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:10.743129    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:10.881885 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:10.881850    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:10.882277 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:10.881995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:10.882277 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:10.882002    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 14:43:10.882277 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:10.882140    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:18.882125202 +0000 UTC m=+69.359185137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : configmap "kube-rbac-proxy" not found
Sep 10 14:43:10.882277 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:10.882145    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:10.882277 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:10.882201    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:18.882186558 +0000 UTC m=+69.359246500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:11.132277 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:11.132188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:11.732558 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:11.732516    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:12.245866 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:12.245839    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kfmvq"
Sep 10 14:43:12.345389 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:12.345360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:12.537289 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:12.537205    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:12.729945 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:12.729912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:12.928518 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:12.928455    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:13.215582 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:13.215456    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:13.737133 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:13.737102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:13.838348 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:13.838319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:13.938376 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:13.938342    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:14.131677 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.131597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:14.331628 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.331598    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:14.399000 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.398968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:14.719122 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.719048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:43:14.721355 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.721326    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 14:43:14.730899 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.730863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:14.732050 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.732028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e30e502c-ff23-4cad-99f1-5c1b501b3fb4-original-pull-secret\") pod \"global-pull-secret-syncer-g9xjj\" (UID: \"e30e502c-ff23-4cad-99f1-5c1b501b3fb4\") " pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:43:14.819616 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.819585    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:43:14.822162 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.822146    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 14:43:14.826054 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.826040    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-g9xjj"
Sep 10 14:43:14.830160 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:14.830144    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 14:43:14.830224 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:14.830197    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs podName:34c67475-95fc-480f-b867-0549df8aff57 nodeName:}" failed. No retries permitted until 2026-09-10 14:44:18.830181669 +0000 UTC m=+129.307241592 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs") pod "network-metrics-daemon-lk75l" (UID: "34c67475-95fc-480f-b867-0549df8aff57") : secret "metrics-daemon-secret" not found
Sep 10 14:43:14.920685 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.920653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:43:14.923929 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:14.923901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztr8k\" (UniqueName: \"kubernetes.io/projected/66388537-9e27-479d-9381-8a8116b234ee-kube-api-access-ztr8k\") pod \"network-check-target-mdn58\" (UID: \"66388537-9e27-479d-9381-8a8116b234ee\") " pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:43:15.075409 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.075330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:15.143736 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.143709    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c8srj\""
Sep 10 14:43:15.152401 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.152384    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:43:15.175947 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.175919    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:15.179888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.179861    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-g9xjj"]
Sep 10 14:43:15.183330 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:43:15.183308    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30e502c_ff23_4cad_99f1_5c1b501b3fb4.slice/crio-7bf490635165dfe079744a8c50f2ed7968e2d4699566c772236cd1e293324691 WatchSource:0}: Error finding container 7bf490635165dfe079744a8c50f2ed7968e2d4699566c772236cd1e293324691: Status 404 returned error can't find the container with id 7bf490635165dfe079744a8c50f2ed7968e2d4699566c772236cd1e293324691
Sep 10 14:43:15.391173 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.391145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g9xjj" event={"ID":"e30e502c-ff23-4cad-99f1-5c1b501b3fb4","Type":"ContainerStarted","Data":"7bf490635165dfe079744a8c50f2ed7968e2d4699566c772236cd1e293324691"}
Sep 10 14:43:15.449634 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.449598    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdn58"]
Sep 10 14:43:15.453134 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:43:15.453112    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66388537_9e27_479d_9381_8a8116b234ee.slice/crio-f8206adad651837b580a432db33bf8f2cf6a67c8d984a780ba8497b73877cc7c WatchSource:0}: Error finding container f8206adad651837b580a432db33bf8f2cf6a67c8d984a780ba8497b73877cc7c: Status 404 returned error can't find the container with id f8206adad651837b580a432db33bf8f2cf6a67c8d984a780ba8497b73877cc7c
Sep 10 14:43:15.527376 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.527350    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:15.731262 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.731179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:15.953912 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:15.953875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:16.250487 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:16.250440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:16.349680 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:16.349643    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:16.397259 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:16.397224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdn58" event={"ID":"66388537-9e27-479d-9381-8a8116b234ee","Type":"ContainerStarted","Data":"7fbbf0a5e86c49df76bf8d478daeef9aae6809a0ba03d9f6cf62d4d5fa4ab07d"}
Sep 10 14:43:16.397437 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:16.397277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdn58" event={"ID":"66388537-9e27-479d-9381-8a8116b234ee","Type":"ContainerStarted","Data":"f8206adad651837b580a432db33bf8f2cf6a67c8d984a780ba8497b73877cc7c"}
Sep 10 14:43:16.397437 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:16.397352    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:43:16.411716 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:16.411666    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mdn58" podStartSLOduration=66.411645422 podStartE2EDuration="1m6.411645422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:16.410541603 +0000 UTC m=+66.887601550" watchObservedRunningTime="2026-09-10 14:43:16.411645422 +0000 UTC m=+66.888705373"
Sep 10 14:43:16.414806 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:16.414775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:16.725375 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:16.725318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:17.526378 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:17.526318    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:17.925524 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:17.925488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:18.125158 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:18.125123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:18.326909 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:18.326821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:18.783477 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:18.783439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:18.959356 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:18.959319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:18.959560 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:18.959421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:18.959560 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:18.959500    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:18.959689 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:18.959594    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls podName:f03b9612-34b5-4495-b105-777d664603fa nodeName:}" failed. No retries permitted until 2026-09-10 14:43:34.959574647 +0000 UTC m=+85.436634578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5249m" (UID: "f03b9612-34b5-4495-b105-777d664603fa") : secret "insights-runtime-extractor-tls" not found
Sep 10 14:43:18.960039 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:18.960021    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f03b9612-34b5-4495-b105-777d664603fa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:19.367298 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.367264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:19.764901 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.764866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:19.868138 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.868106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:43:19.868616 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.868143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:19.868616 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.868250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:19.868616 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.868287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:43:19.868616 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.868345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:43:19.868852 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.868796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b16477-9d87-43a7-9f53-2f5621126de6-service-ca-bundle\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:19.870837 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.870809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9b16477-9d87-43a7-9f53-2f5621126de6-metrics-certs\") pod \"router-default-59ccbd4756-2q8pl\" (UID: \"f9b16477-9d87-43a7-9f53-2f5621126de6\") " pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:19.870952 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.870933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9da82716-f9d8-443f-bb14-0552cd0c3a7a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-f4qtm\" (UID: \"9da82716-f9d8-443f-bb14-0552cd0c3a7a\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:43:19.870995 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.870945    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"image-registry-64fb7c8dd9-qjmvl\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") " pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:43:19.871027 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.870992    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e95b1904-9052-4ab0-bbb1-7ec5cdf770c7-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-mfp2b\" (UID: \"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:43:19.940767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.940737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:19.973890 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.973867    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7kccf\""
Sep 10 14:43:19.982742 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.982716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:19.984999 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.984980    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-s7zkb\""
Sep 10 14:43:19.993785 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:19.993760    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"
Sep 10 14:43:20.004035 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.004014    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gb7kw\""
Sep 10 14:43:20.012172 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.012154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:43:20.065004 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.064969    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bf7bc\""
Sep 10 14:43:20.067823 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.067591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"
Sep 10 14:43:20.070588 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.069972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:43:20.073838 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.073797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bab8655-a495-49a2-80e3-9a2177570f06-metrics-tls\") pod \"dns-default-mkrsh\" (UID: \"0bab8655-a495-49a2-80e3-9a2177570f06\") " pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:43:20.131888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.131831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:20.254072 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.254045    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mkqg5\""
Sep 10 14:43:20.262444 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.262349    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:43:20.263745 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.263720    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59ccbd4756-2q8pl"]
Sep 10 14:43:20.266713 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:43:20.266687    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b16477_9d87_43a7_9f53_2f5621126de6.slice/crio-2dafed0759e1027d6eae0dffae3f706afafcef9e18b89fe9628e65195eb42ab8 WatchSource:0}: Error finding container 2dafed0759e1027d6eae0dffae3f706afafcef9e18b89fe9628e65195eb42ab8: Status 404 returned error can't find the container with id 2dafed0759e1027d6eae0dffae3f706afafcef9e18b89fe9628e65195eb42ab8
Sep 10 14:43:20.283398 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.283373    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm"]
Sep 10 14:43:20.286614 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:43:20.286591    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da82716_f9d8_443f_bb14_0552cd0c3a7a.slice/crio-79e228bf86552ea59d1c981331c1701252d82d677bc032d67a40159865d16e1b WatchSource:0}: Error finding container 79e228bf86552ea59d1c981331c1701252d82d677bc032d67a40159865d16e1b: Status 404 returned error can't find the container with id 79e228bf86552ea59d1c981331c1701252d82d677bc032d67a40159865d16e1b
Sep 10 14:43:20.316016 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.314480    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"]
Sep 10 14:43:20.323076 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:43:20.321183    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b2cf5a_81f8_481d_aa65_92277f3c1328.slice/crio-02a11c48320894303f3e9d74b17896f8b4667b050d023b7d094476b08253fdf3 WatchSource:0}: Error finding container 02a11c48320894303f3e9d74b17896f8b4667b050d023b7d094476b08253fdf3: Status 404 returned error can't find the container with id 02a11c48320894303f3e9d74b17896f8b4667b050d023b7d094476b08253fdf3
Sep 10 14:43:20.350843 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.350816    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b"]
Sep 10 14:43:20.412343 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.412224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl" event={"ID":"c9b2cf5a-81f8-481d-aa65-92277f3c1328","Type":"ContainerStarted","Data":"ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9"}
Sep 10 14:43:20.412343 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.412273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl" event={"ID":"c9b2cf5a-81f8-481d-aa65-92277f3c1328","Type":"ContainerStarted","Data":"02a11c48320894303f3e9d74b17896f8b4667b050d023b7d094476b08253fdf3"}
Sep 10 14:43:20.412565 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.412381    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:43:20.413766 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.413743    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-g9xjj" event={"ID":"e30e502c-ff23-4cad-99f1-5c1b501b3fb4","Type":"ContainerStarted","Data":"e27ac8d55d0b7a04c4fa7212faf4034ad4b0c338f5278679e3d52f8a4137e86c"}
Sep 10 14:43:20.414917 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.414894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b" event={"ID":"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7","Type":"ContainerStarted","Data":"9bfa16f933ae7e68f2ab0b7b38836247bd28ab2c9ff5853b7832676df7944c13"}
Sep 10 14:43:20.416032 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.416007    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm" event={"ID":"9da82716-f9d8-443f-bb14-0552cd0c3a7a","Type":"ContainerStarted","Data":"79e228bf86552ea59d1c981331c1701252d82d677bc032d67a40159865d16e1b"}
Sep 10 14:43:20.417540 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.417517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59ccbd4756-2q8pl" event={"ID":"f9b16477-9d87-43a7-9f53-2f5621126de6","Type":"ContainerStarted","Data":"3541658b25ca8469b838adfe598359b863d884a96a4288701ac9602783209e77"}
Sep 10 14:43:20.417638 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.417558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59ccbd4756-2q8pl" event={"ID":"f9b16477-9d87-43a7-9f53-2f5621126de6","Type":"ContainerStarted","Data":"2dafed0759e1027d6eae0dffae3f706afafcef9e18b89fe9628e65195eb42ab8"}
Sep 10 14:43:20.435453 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.435413    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl" podStartSLOduration=59.435398683 podStartE2EDuration="59.435398683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:20.434593404 +0000 UTC m=+70.911653350" watchObservedRunningTime="2026-09-10 14:43:20.435398683 +0000 UTC m=+70.912458617"
Sep 10 14:43:20.451463 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.451426    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-g9xjj" podStartSLOduration=66.25198883 podStartE2EDuration="1m10.451414266s" totalImagesPullingTime="4.199425436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:20.45076814 +0000 UTC m=+70.927828081" watchObservedRunningTime="2026-09-10 14:43:20.451414266 +0000 UTC m=+70.928474209"
Sep 10 14:43:20.472199 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.472157    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59ccbd4756-2q8pl" podStartSLOduration=68.472143284 podStartE2EDuration="1m8.472143284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:20.471332863 +0000 UTC m=+70.948392803" watchObservedRunningTime="2026-09-10 14:43:20.472143284 +0000 UTC m=+70.949203228"
Sep 10 14:43:20.539380 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.539125    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mkrsh"]
Sep 10 14:43:20.541712 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:43:20.541685    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bab8655_a495_49a2_80e3_9a2177570f06.slice/crio-4d0fbafecf4d616fd8fb27b34888448fa9b9e011b8bcbcd0b44f72c3b42a5ba2 WatchSource:0}: Error finding container 4d0fbafecf4d616fd8fb27b34888448fa9b9e011b8bcbcd0b44f72c3b42a5ba2: Status 404 returned error can't find the container with id 4d0fbafecf4d616fd8fb27b34888448fa9b9e011b8bcbcd0b44f72c3b42a5ba2
Sep 10 14:43:20.558423 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.558396    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:20.983998 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.983958    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:20.989032 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:20.989002    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:21.425637 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:21.425601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mkrsh" event={"ID":"0bab8655-a495-49a2-80e3-9a2177570f06","Type":"ContainerStarted","Data":"4d0fbafecf4d616fd8fb27b34888448fa9b9e011b8bcbcd0b44f72c3b42a5ba2"}
Sep 10 14:43:21.426096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:21.426024    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:21.427307 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:21.427287    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59ccbd4756-2q8pl"
Sep 10 14:43:22.026768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:22.026733    2563 scope.go:117] "RemoveContainer" containerID="556420248e3e360c56055e29d09f359d0820b5642e13edeee54da9f29a610fee"
Sep 10 14:43:23.253030 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.252995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:23.356725 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.356687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:23.438454 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.438412    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b" event={"ID":"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7","Type":"ContainerStarted","Data":"b0a6cc00be00cb78116c5094fcf4a5a4cdc54eaee84e1dd42f1bdecf5f513259"}
Sep 10 14:43:23.438673 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.438541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b" event={"ID":"e95b1904-9052-4ab0-bbb1-7ec5cdf770c7","Type":"ContainerStarted","Data":"abd260a46cb72bbfce917c8100a60c6a91ee4aa7dd64d82fe288e1d189c659d6"}
Sep 10 14:43:23.440065 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.440038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm" event={"ID":"9da82716-f9d8-443f-bb14-0552cd0c3a7a","Type":"ContainerStarted","Data":"61c591f57cc43e0a51caac881cf8a10e5683bbfc341c02ab368884914a49000a"}
Sep 10 14:43:23.441977 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.441952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 14:43:23.442080 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.442060    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8wpzz" event={"ID":"2a2b0239-057f-4348-8709-2406a8d82fae","Type":"ContainerStarted","Data":"742a5ff4909668381398f97d778d51a88fe2f0dd50f107259f61df982befd4fb"}
Sep 10 14:43:23.442352 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.442327    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:43:23.444040 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.444015    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mkrsh" event={"ID":"0bab8655-a495-49a2-80e3-9a2177570f06","Type":"ContainerStarted","Data":"e6f6edcb8c36fcf1f4d66e6294ce2f0b6f892f68e3b8bee66cdc62c3a82749d9"}
Sep 10 14:43:23.444150 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.444065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mkrsh" event={"ID":"0bab8655-a495-49a2-80e3-9a2177570f06","Type":"ContainerStarted","Data":"33cd9b7a96f18cc0fe66def263f3fa65953745e040cf44b1f74c44b7ad50ef3a"}
Sep 10 14:43:23.447346 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.447320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-8wpzz"
Sep 10 14:43:23.454229 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.454181    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-mfp2b" podStartSLOduration=58.927266237 podStartE2EDuration="1m1.454163559s" totalImagesPullingTime="2.526897322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:23.45269024 +0000 UTC m=+73.929750182" watchObservedRunningTime="2026-09-10 14:43:23.454163559 +0000 UTC m=+73.931223504"
Sep 10 14:43:23.454964 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.454939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:23.473519 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.473462    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mkrsh" podStartSLOduration=34.106774632 podStartE2EDuration="36.473449122s" totalImagesPullingTime="2.36667449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:23.472617864 +0000 UTC m=+73.949677809" watchObservedRunningTime="2026-09-10 14:43:23.473449122 +0000 UTC m=+73.950509066"
Sep 10 14:43:23.488200 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.488155    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-8wpzz" podStartSLOduration=48.611766657 podStartE2EDuration="1m1.48814023s" totalImagesPullingTime="12.876373573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:23.486655965 +0000 UTC m=+73.963715909" watchObservedRunningTime="2026-09-10 14:43:23.48814023 +0000 UTC m=+73.965200175"
Sep 10 14:43:23.502596 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.502535    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-f4qtm" podStartSLOduration=68.882760132 podStartE2EDuration="1m11.502519286s" totalImagesPullingTime="2.619759154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:23.499676958 +0000 UTC m=+73.976736903" watchObservedRunningTime="2026-09-10 14:43:23.502519286 +0000 UTC m=+73.979579227"
Sep 10 14:43:23.561996 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.561956    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:23.662857 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.662811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:23.712949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.712877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:23.849637 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:23.849596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:24.244806 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:24.244767    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:24.314115 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:24.314076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:24.448563 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:24.448535    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:43:25.092537 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:25.092482    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:25.195410 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:25.195380    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:25.284471 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:25.284431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:25.414053 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:25.414018    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:25.513642 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:25.513610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:25.642817 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:25.642777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:25.935491 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:25.935449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:26.424805 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:26.424766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:26.486098 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:26.486063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:26.802002 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:26.801913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:26.859023 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:26.858986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:27.455986 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:27.455953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:27.534878 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:27.534847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:28.168724 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:28.168686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:31.188653 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:31.188614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:31.429843 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:31.429810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:31.496221 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:31.496140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:32.638496 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.638362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:32.832455 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.832419    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:32.878394 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.878363    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bmp7j"]
Sep 10 14:43:32.902468 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.902407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.904466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.904443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 14:43:32.904668 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.904650    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 14:43:32.904753 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.904738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vfxxz\""
Sep 10 14:43:32.905018 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.905002    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 14:43:32.905092 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.905031    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 14:43:32.933128 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.933098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:32.978786 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.978761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-sys\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.978786 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.978788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-tls\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.978950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.978822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-accelerators-collector-config\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.978950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.978904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgfbg\" (UniqueName: \"kubernetes.io/projected/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-kube-api-access-tgfbg\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.979032 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.978949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-textfile\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.979032 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.978989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-metrics-client-ca\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.979095 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.979052    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-root\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.979130 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.979092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.979163 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.979137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-wtmp\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:32.998813 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:32.998791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:33.080047 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-wtmp\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080193 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-sys\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080193 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-tls\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080193 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080099    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-accelerators-collector-config\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080193 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgfbg\" (UniqueName: \"kubernetes.io/projected/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-kube-api-access-tgfbg\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080193 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-textfile\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080193 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-sys\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080193 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-metrics-client-ca\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080502 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080204    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-wtmp\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080502 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-root\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080502 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080264    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080502 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:33.080208    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 14:43:33.080502 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080289    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-root\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080502 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:43:33.080349    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-tls podName:b2eeea63-9b0f-4883-ae9a-9636e5ffe157 nodeName:}" failed. No retries permitted until 2026-09-10 14:43:33.580328315 +0000 UTC m=+84.057388239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-tls") pod "node-exporter-bmp7j" (UID: "b2eeea63-9b0f-4883-ae9a-9636e5ffe157") : secret "node-exporter-tls" not found
Sep 10 14:43:33.080742 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080537    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-textfile\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-accelerators-collector-config\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.080810 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.080771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-metrics-client-ca\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.082448 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.082425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.088322 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.088302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgfbg\" (UniqueName: \"kubernetes.io/projected/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-kube-api-access-tgfbg\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.099337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.099315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:33.205849 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.205760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:33.306608 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.306574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:33.406418 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.406384    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:33.506763 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.506680    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:33.584442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.584408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-tls\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.586793 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.586770    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2eeea63-9b0f-4883-ae9a-9636e5ffe157-node-exporter-tls\") pod \"node-exporter-bmp7j\" (UID: \"b2eeea63-9b0f-4883-ae9a-9636e5ffe157\") " pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.606771 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.606728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:33.707190 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.707155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:33.811718 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.811654    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bmp7j"
Sep 10 14:43:33.833779 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.833749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:33.853172 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:43:33.853143    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2eeea63_9b0f_4883_ae9a_9636e5ffe157.slice/crio-a40cf7c436f0dcfb1bf93a312666048e8da0665a43d145223a28fd578d2852cb WatchSource:0}: Error finding container a40cf7c436f0dcfb1bf93a312666048e8da0665a43d145223a28fd578d2852cb: Status 404 returned error can't find the container with id a40cf7c436f0dcfb1bf93a312666048e8da0665a43d145223a28fd578d2852cb
Sep 10 14:43:33.921394 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:33.921360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:34.135292 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:34.135202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:34.454808 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:34.454782    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mkrsh"
Sep 10 14:43:34.486953 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:34.486917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bmp7j" event={"ID":"b2eeea63-9b0f-4883-ae9a-9636e5ffe157","Type":"ContainerStarted","Data":"a40cf7c436f0dcfb1bf93a312666048e8da0665a43d145223a28fd578d2852cb"}
Sep 10 14:43:34.997369 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:34.997335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:34.999454 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:34.999430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f03b9612-34b5-4495-b105-777d664603fa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5249m\" (UID: \"f03b9612-34b5-4495-b105-777d664603fa\") " pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:35.134060 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.134025    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:35.180969 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.180903    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fmx4m\""
Sep 10 14:43:35.189567 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.189542    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5249m"
Sep 10 14:43:35.235376 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.235343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:35.289746 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.289720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:35.461405 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.461240    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5249m"]
Sep 10 14:43:35.464070 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:43:35.464044    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03b9612_34b5_4495_b105_777d664603fa.slice/crio-ef69025351116d2f2ff3281e2bbd01b218e695fb2267d85ef68daa907c6fb235 WatchSource:0}: Error finding container ef69025351116d2f2ff3281e2bbd01b218e695fb2267d85ef68daa907c6fb235: Status 404 returned error can't find the container with id ef69025351116d2f2ff3281e2bbd01b218e695fb2267d85ef68daa907c6fb235
Sep 10 14:43:35.493097 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.493064    2563 generic.go:356] "Generic (PLEG): container finished" podID="b2eeea63-9b0f-4883-ae9a-9636e5ffe157" containerID="b67c734cc6c796a45ca285489c9801af5299809fb996946c204477d56a44ad12" exitCode=0
Sep 10 14:43:35.493202 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.493115    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bmp7j" event={"ID":"b2eeea63-9b0f-4883-ae9a-9636e5ffe157","Type":"ContainerDied","Data":"b67c734cc6c796a45ca285489c9801af5299809fb996946c204477d56a44ad12"}
Sep 10 14:43:35.494635 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.494616    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5249m" event={"ID":"f03b9612-34b5-4495-b105-777d664603fa","Type":"ContainerStarted","Data":"ef69025351116d2f2ff3281e2bbd01b218e695fb2267d85ef68daa907c6fb235"}
Sep 10 14:43:35.827845 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.827769    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:35.881294 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:35.881258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:36.093138 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.093052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:36.188912 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.188869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:36.401332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.401304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:36.499553 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.499481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:36.501677 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.501624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bmp7j" event={"ID":"b2eeea63-9b0f-4883-ae9a-9636e5ffe157","Type":"ContainerStarted","Data":"ae761be1cf69fc8d11936493d612205f2f237967486e169a114162a57752696b"}
Sep 10 14:43:36.501677 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.501673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bmp7j" event={"ID":"b2eeea63-9b0f-4883-ae9a-9636e5ffe157","Type":"ContainerStarted","Data":"5aa35877e1029f8921edc36212d63dc7581e62bfb6c558e41eb9cae5ebcb2f94"}
Sep 10 14:43:36.503500 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.503479    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5249m" event={"ID":"f03b9612-34b5-4495-b105-777d664603fa","Type":"ContainerStarted","Data":"a0bb64478bd7efc2be0115a041fbda812e9bb51f7fd587e50c00639a481df899"}
Sep 10 14:43:36.503617 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.503533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5249m" event={"ID":"f03b9612-34b5-4495-b105-777d664603fa","Type":"ContainerStarted","Data":"ae2ea20460ec036a5bea499a339ecebc3fd8a707fc34f68dda58c1cd961f5287"}
Sep 10 14:43:36.518993 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.518954    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bmp7j" podStartSLOduration=3.412016268 podStartE2EDuration="4.518936136s" totalImagesPullingTime="1.043027476s" totalInitContainerRuntime="63.892392ms" isStatefulPod=true podCreationTimestamp="2026-09-10 14:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:36.517873243 +0000 UTC m=+86.994933187" watchObservedRunningTime="2026-09-10 14:43:36.518936136 +0000 UTC m=+86.995996082"
Sep 10 14:43:36.651023 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:36.650984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:37.509322 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:37.509289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5249m" event={"ID":"f03b9612-34b5-4495-b105-777d664603fa","Type":"ContainerStarted","Data":"e7992076e4bdfce46d593cb2be1f9be8bdc8dc7b48910142eee47ec6bdab6f90"}
Sep 10 14:43:37.526029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:37.525988    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5249m" podStartSLOduration=32.732775006 podStartE2EDuration="34.525972424s" totalImagesPullingTime="1.793197418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:43:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:43:37.525635143 +0000 UTC m=+88.002695108" watchObservedRunningTime="2026-09-10 14:43:37.525972424 +0000 UTC m=+88.003032392"
Sep 10 14:43:37.627578 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:37.627489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:37.686237 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:37.686197    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:37.744414 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:37.744377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:37.954064 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:37.954034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:39.106432 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:39.106370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:40.016885 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:40.016845    2563 patch_prober.go:28] interesting pod/image-registry-64fb7c8dd9-qjmvl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 14:43:40.017059 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:40.016899    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl" podUID="c9b2cf5a-81f8-481d-aa65-92277f3c1328" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 14:43:41.430889 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:41.430858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:43:41.472923 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:41.472895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:41.573001 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:41.572974    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:41.623746 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:41.623716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:41.808786 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:41.808704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:41.861374 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:41.861338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:43.580693 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:43.580650    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:43.646139 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:43.646088    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:44.729934 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:44.729899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:44.779948 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:44.779914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:46.326091 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:46.326063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:46.377324 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:46.377286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:47.301269 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:47.301239    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"]
Sep 10 14:43:47.383391 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:47.383365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:47.402571 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:47.402549    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mdn58"
Sep 10 14:43:47.484369 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:47.484326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:51.119029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:51.118983    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:51.220314 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:51.220279    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:52.582742 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:52.582711    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:52.679680 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:52.679645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:54.692777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:54.692734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:54.791033 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:54.790998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:54.892225 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:54.892189    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:55.406209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:55.406167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:55.506284 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:55.506236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:56.797395 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:56.797363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:43:56.850669 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:43:56.850624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:07.621788 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:07.621701    2563 generic.go:356] "Generic (PLEG): container finished" podID="2c545044-fe16-41b2-8245-4854e96cc5d3" containerID="259ecf5c5e99bfb9202dd5528273349b11465ff3b7baa18a4de7854e90b0cb35" exitCode=0
Sep 10 14:44:07.621788 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:07.621751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr" event={"ID":"2c545044-fe16-41b2-8245-4854e96cc5d3","Type":"ContainerDied","Data":"259ecf5c5e99bfb9202dd5528273349b11465ff3b7baa18a4de7854e90b0cb35"}
Sep 10 14:44:07.622186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:07.622072    2563 scope.go:117] "RemoveContainer" containerID="259ecf5c5e99bfb9202dd5528273349b11465ff3b7baa18a4de7854e90b0cb35"
Sep 10 14:44:08.626882 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:08.626848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-z9mrr" event={"ID":"2c545044-fe16-41b2-8245-4854e96cc5d3","Type":"ContainerStarted","Data":"382d4c7a9341197e0d035b95d642d69300e7756d7d2315e13fcd7cff0a47262b"}
Sep 10 14:44:11.781363 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:11.781323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:11.830558 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:11.830530    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:12.320586 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.320521    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl" podUID="c9b2cf5a-81f8-481d-aa65-92277f3c1328" containerName="registry" containerID="cri-o://ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9" gracePeriod=30
Sep 10 14:44:12.560068 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.560045    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:44:12.617186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617115    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-image-registry-private-configuration\") pod \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") "
Sep 10 14:44:12.617186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617148    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-installation-pull-secrets\") pod \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") "
Sep 10 14:44:12.617186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617174    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-certificates\") pod \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") "
Sep 10 14:44:12.617450 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-bound-sa-token\") pod \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") "
Sep 10 14:44:12.617450 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kmpm\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-kube-api-access-8kmpm\") pod \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") "
Sep 10 14:44:12.617450 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") pod \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") "
Sep 10 14:44:12.617450 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9b2cf5a-81f8-481d-aa65-92277f3c1328-ca-trust-extracted\") pod \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") "
Sep 10 14:44:12.617450 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-trusted-ca\") pod \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\" (UID: \"c9b2cf5a-81f8-481d-aa65-92277f3c1328\") "
Sep 10 14:44:12.617781 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617733    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c9b2cf5a-81f8-481d-aa65-92277f3c1328" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:44:12.617910 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.617880    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c9b2cf5a-81f8-481d-aa65-92277f3c1328" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:44:12.619994 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.619935    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c9b2cf5a-81f8-481d-aa65-92277f3c1328" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:44:12.620210 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.620093    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c9b2cf5a-81f8-481d-aa65-92277f3c1328" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:44:12.620299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.620267    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c9b2cf5a-81f8-481d-aa65-92277f3c1328" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:44:12.620397 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.620306    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c9b2cf5a-81f8-481d-aa65-92277f3c1328" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:44:12.620397 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.620315    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-kube-api-access-8kmpm" (OuterVolumeSpecName: "kube-api-access-8kmpm") pod "c9b2cf5a-81f8-481d-aa65-92277f3c1328" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328"). InnerVolumeSpecName "kube-api-access-8kmpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:44:12.628832 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.628801    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b2cf5a-81f8-481d-aa65-92277f3c1328-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c9b2cf5a-81f8-481d-aa65-92277f3c1328" (UID: "c9b2cf5a-81f8-481d-aa65-92277f3c1328"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:44:12.642025 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.641998    2563 generic.go:356] "Generic (PLEG): container finished" podID="c9b2cf5a-81f8-481d-aa65-92277f3c1328" containerID="ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9" exitCode=0
Sep 10 14:44:12.642131 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.642040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"
Sep 10 14:44:12.642131 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.642052    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl" event={"ID":"c9b2cf5a-81f8-481d-aa65-92277f3c1328","Type":"ContainerDied","Data":"ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9"}
Sep 10 14:44:12.642131 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.642103    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64fb7c8dd9-qjmvl" event={"ID":"c9b2cf5a-81f8-481d-aa65-92277f3c1328","Type":"ContainerDied","Data":"02a11c48320894303f3e9d74b17896f8b4667b050d023b7d094476b08253fdf3"}
Sep 10 14:44:12.642131 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.642130    2563 scope.go:117] "RemoveContainer" containerID="ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9"
Sep 10 14:44:12.652394 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.652371    2563 scope.go:117] "RemoveContainer" containerID="ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9"
Sep 10 14:44:12.652742 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:44:12.652716    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9\": container with ID starting with ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9 not found: ID does not exist" containerID="ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9"
Sep 10 14:44:12.652836 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.652750    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9"} err="failed to get container status \"ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9\": rpc error: code = NotFound desc = could not find container \"ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9\": container with ID starting with ed8bac7f6bc74fc5d581ba4c2c3ba8e9c62dc0cf618a796993f5c8b91d089ba9 not found: ID does not exist"
Sep 10 14:44:12.664791 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.664763    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"]
Sep 10 14:44:12.668236 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.668215    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64fb7c8dd9-qjmvl"]
Sep 10 14:44:12.718450 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.718426    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-image-registry-private-configuration\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:44:12.718450 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.718446    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c9b2cf5a-81f8-481d-aa65-92277f3c1328-installation-pull-secrets\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:44:12.718612 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.718457    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-certificates\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:44:12.718612 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.718466    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-bound-sa-token\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:44:12.718612 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.718475    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kmpm\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-kube-api-access-8kmpm\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:44:12.718612 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.718484    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c9b2cf5a-81f8-481d-aa65-92277f3c1328-registry-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:44:12.718612 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.718492    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c9b2cf5a-81f8-481d-aa65-92277f3c1328-ca-trust-extracted\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:44:12.718612 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.718500    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c9b2cf5a-81f8-481d-aa65-92277f3c1328-trusted-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:44:12.808770 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.808746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:12.861730 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:12.861697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:14.030497 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:14.030460    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9b2cf5a-81f8-481d-aa65-92277f3c1328" path="/var/lib/kubelet/pods/c9b2cf5a-81f8-481d-aa65-92277f3c1328/volumes"
Sep 10 14:44:18.423248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:18.423210    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" podUID="621cf180-3ec9-49bd-b66e-e0da3ba2f968" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 14:44:18.863887 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:18.863797    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:44:18.866171 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:18.866140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34c67475-95fc-480f-b867-0549df8aff57-metrics-certs\") pod \"network-metrics-daemon-lk75l\" (UID: \"34c67475-95fc-480f-b867-0549df8aff57\") " pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:44:19.038930 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:19.038900    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gnf6m\""
Sep 10 14:44:19.046917 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:19.046893    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lk75l"
Sep 10 14:44:19.321226 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:19.321199    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lk75l"]
Sep 10 14:44:19.323710 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:44:19.323681    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c67475_95fc_480f_b867_0549df8aff57.slice/crio-877243e15dd78f6146ee14c78c66de8e1a28d6a3a90c71f6dcc83b00fa46ebd3 WatchSource:0}: Error finding container 877243e15dd78f6146ee14c78c66de8e1a28d6a3a90c71f6dcc83b00fa46ebd3: Status 404 returned error can't find the container with id 877243e15dd78f6146ee14c78c66de8e1a28d6a3a90c71f6dcc83b00fa46ebd3
Sep 10 14:44:19.674548 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:19.674470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lk75l" event={"ID":"34c67475-95fc-480f-b867-0549df8aff57","Type":"ContainerStarted","Data":"877243e15dd78f6146ee14c78c66de8e1a28d6a3a90c71f6dcc83b00fa46ebd3"}
Sep 10 14:44:20.681110 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:20.681026    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lk75l" event={"ID":"34c67475-95fc-480f-b867-0549df8aff57","Type":"ContainerStarted","Data":"2d54787264a4f223f6d65fe8e9a6762112565baaa9a5e91cc080b7f50a8b7541"}
Sep 10 14:44:20.681545 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:20.681129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lk75l" event={"ID":"34c67475-95fc-480f-b867-0549df8aff57","Type":"ContainerStarted","Data":"c80e435f6a8090021e82d85bfbabad28e01237e39dbce9d1db83a0b5e58274fc"}
Sep 10 14:44:20.696748 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:20.696689    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lk75l" podStartSLOduration=129.584399053 podStartE2EDuration="2m10.696666099s" totalImagesPullingTime="1.112267046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:44:20.694085819 +0000 UTC m=+131.171145768" watchObservedRunningTime="2026-09-10 14:44:20.696666099 +0000 UTC m=+131.173726044"
Sep 10 14:44:20.780088 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:20.780053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:20.837435 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:20.837391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:27.708242 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:27.708210    2563 generic.go:356] "Generic (PLEG): container finished" podID="809cecbd-174e-4ca6-9a14-aa49acfb7e67" containerID="5eb9f20eda172a0bbc8f74d9c37c3fc7b139b5e93a3a86e5626e8ea00dc2ea31" exitCode=0
Sep 10 14:44:27.708645 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:27.708283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s" event={"ID":"809cecbd-174e-4ca6-9a14-aa49acfb7e67","Type":"ContainerDied","Data":"5eb9f20eda172a0bbc8f74d9c37c3fc7b139b5e93a3a86e5626e8ea00dc2ea31"}
Sep 10 14:44:27.708645 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:27.708620    2563 scope.go:117] "RemoveContainer" containerID="5eb9f20eda172a0bbc8f74d9c37c3fc7b139b5e93a3a86e5626e8ea00dc2ea31"
Sep 10 14:44:28.423555 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:28.423490    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" podUID="621cf180-3ec9-49bd-b66e-e0da3ba2f968" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 14:44:28.716020 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:28.715939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-6tf2s" event={"ID":"809cecbd-174e-4ca6-9a14-aa49acfb7e67","Type":"ContainerStarted","Data":"5ab9b20634dc0698aa71c42114d132c06a20997df9a60c7675eec10e95408047"}
Sep 10 14:44:32.731110 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:32.731075    2563 generic.go:356] "Generic (PLEG): container finished" podID="e3084961-1d48-497e-b105-b9ad92744ee5" containerID="ef78a1ff23014ef8fa0cd9f972fc94dcd8d2a584e518fc9734732e0194475439" exitCode=0
Sep 10 14:44:32.731110 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:32.731119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p" event={"ID":"e3084961-1d48-497e-b105-b9ad92744ee5","Type":"ContainerDied","Data":"ef78a1ff23014ef8fa0cd9f972fc94dcd8d2a584e518fc9734732e0194475439"}
Sep 10 14:44:32.731533 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:32.731396    2563 scope.go:117] "RemoveContainer" containerID="ef78a1ff23014ef8fa0cd9f972fc94dcd8d2a584e518fc9734732e0194475439"
Sep 10 14:44:33.737899 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:33.737869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-t5w7p" event={"ID":"e3084961-1d48-497e-b105-b9ad92744ee5","Type":"ContainerStarted","Data":"33721de39f2b62013cf6bf4ef4c1a6d254542dc38faa080063601883683fb9a4"}
Sep 10 14:44:38.422560 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:38.422485    2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" podUID="621cf180-3ec9-49bd-b66e-e0da3ba2f968" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 14:44:38.422932 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:38.422593    2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l"
Sep 10 14:44:38.423076 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:38.423059    2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"949a92e70fcfd0d37b1ea65dde85ae926ecdfdb76bb7263c4d6e9f0f41740651"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 10 14:44:38.423111 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:38.423096    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" podUID="621cf180-3ec9-49bd-b66e-e0da3ba2f968" containerName="service-proxy" containerID="cri-o://949a92e70fcfd0d37b1ea65dde85ae926ecdfdb76bb7263c4d6e9f0f41740651" gracePeriod=30
Sep 10 14:44:38.758109 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:38.758023    2563 generic.go:356] "Generic (PLEG): container finished" podID="621cf180-3ec9-49bd-b66e-e0da3ba2f968" containerID="949a92e70fcfd0d37b1ea65dde85ae926ecdfdb76bb7263c4d6e9f0f41740651" exitCode=2
Sep 10 14:44:38.758248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:38.758107    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" event={"ID":"621cf180-3ec9-49bd-b66e-e0da3ba2f968","Type":"ContainerDied","Data":"949a92e70fcfd0d37b1ea65dde85ae926ecdfdb76bb7263c4d6e9f0f41740651"}
Sep 10 14:44:38.758248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:38.758149    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57885bc544-p4f8l" event={"ID":"621cf180-3ec9-49bd-b66e-e0da3ba2f968","Type":"ContainerStarted","Data":"1d7843d53b021b22b10e1e1830211a16a4f3264434f39d938909e88c2460e781"}
Sep 10 14:44:48.962834 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:48.962794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:49.063265 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:49.063230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:54.167334 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:54.167302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:54.269419 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:54.269373    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:54.367733 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:54.367695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:54.912413 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:54.912356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:44:54.973597 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:44:54.973565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:00.837478 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:00.837439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:01.171555 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:01.171519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:05.001939 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:05.001853    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:05.060085 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:05.060054    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:05.204484 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:05.204453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:11.811087 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:11.811048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:11.910787 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:11.910750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:38.511312 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:38.511274    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:38.560022 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:38.559980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:45:40.809189 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:45:40.809148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:03.061222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:03.061182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:03.158428 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:03.158397    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:16.096573 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:16.096533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:16.197818 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:16.197783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:29.684159 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:29.684125    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:30.481930 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:30.481891    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:33.501407 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:33.501325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:33.613590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:33.613559    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:34.671449 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:34.671407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:34.763319 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:34.763285    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:35.030979 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:35.030907    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:35.115231 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:35.115194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:37.196688 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:37.196656    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:37.385955 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:37.385924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:40.295097 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:40.295064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:40.353108 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:40.353073    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:43.705246 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:43.705210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:43.800620 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:43.800586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:45.607136 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:45.607090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:50.757251 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:50.757213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:50.859750 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:50.859716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:46:52.936764 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:46:52.936733    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:04.329784 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:04.329749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:05.229388 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.229355    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"]
Sep 10 14:47:05.229810 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.229791    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9b2cf5a-81f8-481d-aa65-92277f3c1328" containerName="registry"
Sep 10 14:47:05.229861 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.229817    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9b2cf5a-81f8-481d-aa65-92277f3c1328" containerName="registry"
Sep 10 14:47:05.229896 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.229886    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9b2cf5a-81f8-481d-aa65-92277f3c1328" containerName="registry"
Sep 10 14:47:05.232709 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.232694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.235188 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.235153    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 14:47:05.235188 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.235179    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 14:47:05.235363 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.235264    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-7smlt\""
Sep 10 14:47:05.235482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.235467    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 14:47:05.235714 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.235688    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 14:47:05.247825 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.247806    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"]
Sep 10 14:47:05.335621 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.335591    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b96e3a59-a1d6-4388-b780-7d0b6e4029db-manifests\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.335621 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.335625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l8t6\" (UniqueName: \"kubernetes.io/projected/b96e3a59-a1d6-4388-b780-7d0b6e4029db-kube-api-access-6l8t6\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.336027 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.335653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b96e3a59-a1d6-4388-b780-7d0b6e4029db-webhook-cert\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.336027 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.335686    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96e3a59-a1d6-4388-b780-7d0b6e4029db-tmp\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.336027 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.335715    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b96e3a59-a1d6-4388-b780-7d0b6e4029db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.436264 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.436229    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b96e3a59-a1d6-4388-b780-7d0b6e4029db-manifests\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.436264 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.436268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l8t6\" (UniqueName: \"kubernetes.io/projected/b96e3a59-a1d6-4388-b780-7d0b6e4029db-kube-api-access-6l8t6\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.436498 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.436294    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b96e3a59-a1d6-4388-b780-7d0b6e4029db-webhook-cert\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.436498 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.436345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96e3a59-a1d6-4388-b780-7d0b6e4029db-tmp\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.436498 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.436394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b96e3a59-a1d6-4388-b780-7d0b6e4029db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.436720 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.436660    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b96e3a59-a1d6-4388-b780-7d0b6e4029db-manifests\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.438539 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.438496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b96e3a59-a1d6-4388-b780-7d0b6e4029db-tmp\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.438801 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.438785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b96e3a59-a1d6-4388-b780-7d0b6e4029db-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.438972 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.438949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b96e3a59-a1d6-4388-b780-7d0b6e4029db-webhook-cert\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.443726 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.443706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l8t6\" (UniqueName: \"kubernetes.io/projected/b96e3a59-a1d6-4388-b780-7d0b6e4029db-kube-api-access-6l8t6\") pod \"opendatahub-operator-controller-manager-68f84f89d7-mdzcx\" (UID: \"b96e3a59-a1d6-4388-b780-7d0b6e4029db\") " pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.542710 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.542636    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:05.822575 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:05.822550    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"]
Sep 10 14:47:05.825524 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:47:05.825476    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96e3a59_a1d6_4388_b780_7d0b6e4029db.slice/crio-f0a05291cd13e115bea2454c0a98f733f7bdd037cb6033c76757c8f2c9c31055 WatchSource:0}: Error finding container f0a05291cd13e115bea2454c0a98f733f7bdd037cb6033c76757c8f2c9c31055: Status 404 returned error can't find the container with id f0a05291cd13e115bea2454c0a98f733f7bdd037cb6033c76757c8f2c9c31055
Sep 10 14:47:06.229822 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:06.229781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:06.298890 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:06.298860    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx" event={"ID":"b96e3a59-a1d6-4388-b780-7d0b6e4029db","Type":"ContainerStarted","Data":"f0a05291cd13e115bea2454c0a98f733f7bdd037cb6033c76757c8f2c9c31055"}
Sep 10 14:47:08.290304 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:08.290224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:08.391445 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:08.391411    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:09.314226 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:09.314129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx" event={"ID":"b96e3a59-a1d6-4388-b780-7d0b6e4029db","Type":"ContainerStarted","Data":"df16a71bc3ebf0fa7723576645fd75731e2432042b3aaf95d4af6ccb3a50d189"}
Sep 10 14:47:09.942611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:09.942581    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 14:47:09.942893 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:09.942873    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 14:47:09.956824 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:09.956803    2563 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 14:47:10.319962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:10.319883    2563 generic.go:356] "Generic (PLEG): container finished" podID="b96e3a59-a1d6-4388-b780-7d0b6e4029db" containerID="df16a71bc3ebf0fa7723576645fd75731e2432042b3aaf95d4af6ccb3a50d189" exitCode=0
Sep 10 14:47:10.319962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:10.319922    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx" event={"ID":"b96e3a59-a1d6-4388-b780-7d0b6e4029db","Type":"ContainerDied","Data":"df16a71bc3ebf0fa7723576645fd75731e2432042b3aaf95d4af6ccb3a50d189"}
Sep 10 14:47:11.326288 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:11.326250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx" event={"ID":"b96e3a59-a1d6-4388-b780-7d0b6e4029db","Type":"ContainerStarted","Data":"dc7e17e96f77a75b9bb559544a529a43bf9a2990cfea8e8fc19131f7bc7d2b53"}
Sep 10 14:47:11.326705 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:11.326427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:11.350865 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:11.350815    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx" podStartSLOduration=1.835894272 podStartE2EDuration="6.350801941s" totalImagesPullingTime="3.507415353s" totalInitContainerRuntime="1.007492316s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:11.349892552 +0000 UTC m=+301.826952499" watchObservedRunningTime="2026-09-10 14:47:11.350801941 +0000 UTC m=+301.827861885"
Sep 10 14:47:21.430999 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:21.430965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:21.535016 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:21.534979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:22.333029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:22.333001    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68f84f89d7-mdzcx"
Sep 10 14:47:24.945711 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:24.945676    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:25.067141 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.067105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:25.402726 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.402691    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:25.442252 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.442219    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"]
Sep 10 14:47:25.453428 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.453403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.455785 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.455759    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 14:47:25.458722 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.458701    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 14:47:25.460674 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.460651    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"]
Sep 10 14:47:25.521318 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.521286    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:25.603488 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.603451    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52d2e320-c84e-4a03-a429-c25bcd89d238-tmp\") pod \"kube-auth-proxy-79b7fd47cd-9h88c\" (UID: \"52d2e320-c84e-4a03-a429-c25bcd89d238\") " pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.603680 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.603559    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d2e320-c84e-4a03-a429-c25bcd89d238-tls-certs\") pod \"kube-auth-proxy-79b7fd47cd-9h88c\" (UID: \"52d2e320-c84e-4a03-a429-c25bcd89d238\") " pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.603680 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.603597    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps5mz\" (UniqueName: \"kubernetes.io/projected/52d2e320-c84e-4a03-a429-c25bcd89d238-kube-api-access-ps5mz\") pod \"kube-auth-proxy-79b7fd47cd-9h88c\" (UID: \"52d2e320-c84e-4a03-a429-c25bcd89d238\") " pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.704660 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.704581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d2e320-c84e-4a03-a429-c25bcd89d238-tls-certs\") pod \"kube-auth-proxy-79b7fd47cd-9h88c\" (UID: \"52d2e320-c84e-4a03-a429-c25bcd89d238\") " pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.704660 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.704628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps5mz\" (UniqueName: \"kubernetes.io/projected/52d2e320-c84e-4a03-a429-c25bcd89d238-kube-api-access-ps5mz\") pod \"kube-auth-proxy-79b7fd47cd-9h88c\" (UID: \"52d2e320-c84e-4a03-a429-c25bcd89d238\") " pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.704660 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.704659    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52d2e320-c84e-4a03-a429-c25bcd89d238-tmp\") pod \"kube-auth-proxy-79b7fd47cd-9h88c\" (UID: \"52d2e320-c84e-4a03-a429-c25bcd89d238\") " pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.706985 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.706961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52d2e320-c84e-4a03-a429-c25bcd89d238-tmp\") pod \"kube-auth-proxy-79b7fd47cd-9h88c\" (UID: \"52d2e320-c84e-4a03-a429-c25bcd89d238\") " pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.707208 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.707189    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d2e320-c84e-4a03-a429-c25bcd89d238-tls-certs\") pod \"kube-auth-proxy-79b7fd47cd-9h88c\" (UID: \"52d2e320-c84e-4a03-a429-c25bcd89d238\") " pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.714445 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.714415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps5mz\" (UniqueName: \"kubernetes.io/projected/52d2e320-c84e-4a03-a429-c25bcd89d238-kube-api-access-ps5mz\") pod \"kube-auth-proxy-79b7fd47cd-9h88c\" (UID: \"52d2e320-c84e-4a03-a429-c25bcd89d238\") " pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:25.764224 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:25.764201    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"
Sep 10 14:47:26.036364 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.036295    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c"]
Sep 10 14:47:26.039233 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:47:26.039204    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d2e320_c84e_4a03_a429_c25bcd89d238.slice/crio-d63657443dad1828db95c7dea1fc200846c55d4ea2667106923a7937c6af98fc WatchSource:0}: Error finding container d63657443dad1828db95c7dea1fc200846c55d4ea2667106923a7937c6af98fc: Status 404 returned error can't find the container with id d63657443dad1828db95c7dea1fc200846c55d4ea2667106923a7937c6af98fc
Sep 10 14:47:26.040767 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.040751    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:47:26.388860 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.388834    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c" event={"ID":"52d2e320-c84e-4a03-a429-c25bcd89d238","Type":"ContainerStarted","Data":"d63657443dad1828db95c7dea1fc200846c55d4ea2667106923a7937c6af98fc"}
Sep 10 14:47:26.897016 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.896970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:26.901998 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.901768    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"]
Sep 10 14:47:26.905254 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.905228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:26.907615 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.907323    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 14:47:26.907991 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.907825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vbqdk\""
Sep 10 14:47:26.908738 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.908717    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 14:47:26.908840 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.908795    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 14:47:26.909123 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.909106    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 14:47:26.919210 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.919189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"]
Sep 10 14:47:26.965080 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:26.964979    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:27.015108 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.015073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7936579f-3755-475c-a158-9496c5226520-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.015299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.015126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.015299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.015161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.015299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.015199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmw9\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-kube-api-access-2kmw9\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.015299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.015232    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.015548 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.015306    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.015548 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.015345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7936579f-3755-475c-a158-9496c5226520-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.116266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.116227    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.116878 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.116281    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.116878 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.116316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmw9\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-kube-api-access-2kmw9\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.116878 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.116351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.116878 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.116414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.116878 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.116444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7936579f-3755-475c-a158-9496c5226520-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.116878 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.116481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7936579f-3755-475c-a158-9496c5226520-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.117397 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.117367    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7936579f-3755-475c-a158-9496c5226520-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.119294 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.119269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7936579f-3755-475c-a158-9496c5226520-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.120071 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.120028    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.120496 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.120456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.120839 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.120797    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.125015 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.124990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.125228 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.125202    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmw9\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-kube-api-access-2kmw9\") pod \"istiod-openshift-gateway-7b7995d5cd-st2lb\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.219719 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.219633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:27.512608 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.512575    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"]
Sep 10 14:47:27.882548 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:27.882460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:28.755558 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:47:28.755528    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7936579f_3755_475c_a158_9496c5226520.slice/crio-769ae26419a5fdc84f62ce876270a94760b9a8599bbca904e2e0668cde76a279 WatchSource:0}: Error finding container 769ae26419a5fdc84f62ce876270a94760b9a8599bbca904e2e0668cde76a279: Status 404 returned error can't find the container with id 769ae26419a5fdc84f62ce876270a94760b9a8599bbca904e2e0668cde76a279
Sep 10 14:47:29.406309 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:29.406272    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb" event={"ID":"7936579f-3755-475c-a158-9496c5226520","Type":"ContainerStarted","Data":"769ae26419a5fdc84f62ce876270a94760b9a8599bbca904e2e0668cde76a279"}
Sep 10 14:47:29.407632 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:29.407603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c" event={"ID":"52d2e320-c84e-4a03-a429-c25bcd89d238","Type":"ContainerStarted","Data":"c7acc8a82000dc160364323fbf4ec33d92f9cf12092c0ea4cc95ae62d9fe88cb"}
Sep 10 14:47:29.423341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:29.423291    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79b7fd47cd-9h88c" podStartSLOduration=1.656236856 podStartE2EDuration="4.423276396s" totalImagesPullingTime="2.76703954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:29.421885785 +0000 UTC m=+319.898945729" watchObservedRunningTime="2026-09-10 14:47:29.423276396 +0000 UTC m=+319.900336339"
Sep 10 14:47:31.181918 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:31.181880    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:47:31.182201 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:31.181945    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:47:31.422212 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:31.422176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb" event={"ID":"7936579f-3755-475c-a158-9496c5226520","Type":"ContainerStarted","Data":"22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5"}
Sep 10 14:47:31.422393 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:31.422301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:31.442961 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:31.442867    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb" podStartSLOduration=3.019002479 podStartE2EDuration="5.442853112s" totalImagesPullingTime="2.423850633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:31.440886893 +0000 UTC m=+321.917946836" watchObservedRunningTime="2026-09-10 14:47:31.442853112 +0000 UTC m=+321.919913056"
Sep 10 14:47:32.429066 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:32.429029    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-st2lb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 14:47:32.429433 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:32.429086    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb" podUID="7936579f-3755-475c-a158-9496c5226520" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 14:47:33.089555 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:33.089494    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:33.189828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:33.189793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:33.739610 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:33.739579    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:33.920897 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:33.920849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:34.233044 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:34.233007    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:35.428187 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:35.428160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:36.225372 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:36.225333    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:36.319599 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:36.319562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:36.419991 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:36.419952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:36.517217 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:36.517123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:36.846848 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:36.846749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:37.370307 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:37.370250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:43.693897 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:43.693856    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:43.792255 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:43.792223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:45.181710 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.181663    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:45.273517 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.273484    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q692j"]
Sep 10 14:47:45.277989 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.277969    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:45.280741 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.280720    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 14:47:45.281068 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.281050    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-pdwfm\""
Sep 10 14:47:45.286990 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.286965    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q692j"]
Sep 10 14:47:45.294745 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.294720    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"]
Sep 10 14:47:45.298352 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.298335    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:47:45.300287 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.300267    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 14:47:45.300585 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.300566    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-22487\""
Sep 10 14:47:45.314144 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.314123    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"]
Sep 10 14:47:45.321890 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.321868    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:45.376700 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.376668    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq87x\" (UniqueName: \"kubernetes.io/projected/f881eda7-2589-4f11-818b-12fbbdc66305-kube-api-access-qq87x\") pod \"llmisvc-controller-manager-777c6bd7d5-ltrn4\" (UID: \"f881eda7-2589-4f11-818b-12fbbdc66305\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:47:45.376849 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.376741    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f881eda7-2589-4f11-818b-12fbbdc66305-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ltrn4\" (UID: \"f881eda7-2589-4f11-818b-12fbbdc66305\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:47:45.376849 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.376769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37acfe05-5214-4191-abb7-741d7d0e46b4-cert\") pod \"kserve-controller-manager-d6f4b65b8-q692j\" (UID: \"37acfe05-5214-4191-abb7-741d7d0e46b4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:45.376849 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.376786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsdkw\" (UniqueName: \"kubernetes.io/projected/37acfe05-5214-4191-abb7-741d7d0e46b4-kube-api-access-vsdkw\") pod \"kserve-controller-manager-d6f4b65b8-q692j\" (UID: \"37acfe05-5214-4191-abb7-741d7d0e46b4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:45.412668 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.412636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:45.477337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.477241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq87x\" (UniqueName: \"kubernetes.io/projected/f881eda7-2589-4f11-818b-12fbbdc66305-kube-api-access-qq87x\") pod \"llmisvc-controller-manager-777c6bd7d5-ltrn4\" (UID: \"f881eda7-2589-4f11-818b-12fbbdc66305\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:47:45.477337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.477326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f881eda7-2589-4f11-818b-12fbbdc66305-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ltrn4\" (UID: \"f881eda7-2589-4f11-818b-12fbbdc66305\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:47:45.477636 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.477356    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37acfe05-5214-4191-abb7-741d7d0e46b4-cert\") pod \"kserve-controller-manager-d6f4b65b8-q692j\" (UID: \"37acfe05-5214-4191-abb7-741d7d0e46b4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:45.477636 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.477381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsdkw\" (UniqueName: \"kubernetes.io/projected/37acfe05-5214-4191-abb7-741d7d0e46b4-kube-api-access-vsdkw\") pod \"kserve-controller-manager-d6f4b65b8-q692j\" (UID: \"37acfe05-5214-4191-abb7-741d7d0e46b4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:45.477636 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:47:45.477482    2563 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 10 14:47:45.477636 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:47:45.477589    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37acfe05-5214-4191-abb7-741d7d0e46b4-cert podName:37acfe05-5214-4191-abb7-741d7d0e46b4 nodeName:}" failed. No retries permitted until 2026-09-10 14:47:45.97756836 +0000 UTC m=+336.454628296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37acfe05-5214-4191-abb7-741d7d0e46b4-cert") pod "kserve-controller-manager-d6f4b65b8-q692j" (UID: "37acfe05-5214-4191-abb7-741d7d0e46b4") : secret "kserve-webhook-server-cert" not found
Sep 10 14:47:45.480084 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.480051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f881eda7-2589-4f11-818b-12fbbdc66305-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-ltrn4\" (UID: \"f881eda7-2589-4f11-818b-12fbbdc66305\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:47:45.491096 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.491070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq87x\" (UniqueName: \"kubernetes.io/projected/f881eda7-2589-4f11-818b-12fbbdc66305-kube-api-access-qq87x\") pod \"llmisvc-controller-manager-777c6bd7d5-ltrn4\" (UID: \"f881eda7-2589-4f11-818b-12fbbdc66305\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:47:45.491220 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.491200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsdkw\" (UniqueName: \"kubernetes.io/projected/37acfe05-5214-4191-abb7-741d7d0e46b4-kube-api-access-vsdkw\") pod \"kserve-controller-manager-d6f4b65b8-q692j\" (UID: \"37acfe05-5214-4191-abb7-741d7d0e46b4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:45.556871 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.556835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:45.608988 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.608955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:47:45.879465 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.879436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"]
Sep 10 14:47:45.881804 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:47:45.881781    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf881eda7_2589_4f11_818b_12fbbdc66305.slice/crio-68f8d540e7e216d4189772505de6a546ebfc52182230446930ddedeb7d81d3c9 WatchSource:0}: Error finding container 68f8d540e7e216d4189772505de6a546ebfc52182230446930ddedeb7d81d3c9: Status 404 returned error can't find the container with id 68f8d540e7e216d4189772505de6a546ebfc52182230446930ddedeb7d81d3c9
Sep 10 14:47:45.980616 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.980584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37acfe05-5214-4191-abb7-741d7d0e46b4-cert\") pod \"kserve-controller-manager-d6f4b65b8-q692j\" (UID: \"37acfe05-5214-4191-abb7-741d7d0e46b4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:45.982912 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:45.982894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37acfe05-5214-4191-abb7-741d7d0e46b4-cert\") pod \"kserve-controller-manager-d6f4b65b8-q692j\" (UID: \"37acfe05-5214-4191-abb7-741d7d0e46b4\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:46.189858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:46.189817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:46.463148 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:46.463108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-q692j"]
Sep 10 14:47:46.465483 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:47:46.465461    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37acfe05_5214_4191_abb7_741d7d0e46b4.slice/crio-df1c3157c5af2a69b096e1508ece9819039cbd6910440987f05ecf76d12584de WatchSource:0}: Error finding container df1c3157c5af2a69b096e1508ece9819039cbd6910440987f05ecf76d12584de: Status 404 returned error can't find the container with id df1c3157c5af2a69b096e1508ece9819039cbd6910440987f05ecf76d12584de
Sep 10 14:47:46.483181 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:46.483152    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:46.488317 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:46.488290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4" event={"ID":"f881eda7-2589-4f11-818b-12fbbdc66305","Type":"ContainerStarted","Data":"68f8d540e7e216d4189772505de6a546ebfc52182230446930ddedeb7d81d3c9"}
Sep 10 14:47:46.489550 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:46.489528    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j" event={"ID":"37acfe05-5214-4191-abb7-741d7d0e46b4","Type":"ContainerStarted","Data":"df1c3157c5af2a69b096e1508ece9819039cbd6910440987f05ecf76d12584de"}
Sep 10 14:47:46.594860 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:46.594798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:46.689316 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:46.689288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:47:47.394349 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.394319    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"]
Sep 10 14:47:47.394736 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.394558    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb" podUID="7936579f-3755-475c-a158-9496c5226520" containerName="discovery" containerID="cri-o://22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5" gracePeriod=30
Sep 10 14:47:47.427502 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.427480    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-st2lb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.24:8080/ready\": dial tcp 10.132.0.24:8080: connect: connection refused" start-of-body=
Sep 10 14:47:47.427603 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.427552    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb" podUID="7936579f-3755-475c-a158-9496c5226520" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.24:8080/ready\": dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 10 14:47:47.663953 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.663917    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:47.693555 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.693485    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7936579f-3755-475c-a158-9496c5226520-istio-csr-ca-configmap\") pod \"7936579f-3755-475c-a158-9496c5226520\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") "
Sep 10 14:47:47.693727 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.693602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-kubeconfig\") pod \"7936579f-3755-475c-a158-9496c5226520\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") "
Sep 10 14:47:47.693727 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.693629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-cacerts\") pod \"7936579f-3755-475c-a158-9496c5226520\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") "
Sep 10 14:47:47.693727 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.693666    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-istio-token\") pod \"7936579f-3755-475c-a158-9496c5226520\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") "
Sep 10 14:47:47.693905 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.693791    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7936579f-3755-475c-a158-9496c5226520-local-certs\") pod \"7936579f-3755-475c-a158-9496c5226520\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") "
Sep 10 14:47:47.693905 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.693837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kmw9\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-kube-api-access-2kmw9\") pod \"7936579f-3755-475c-a158-9496c5226520\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") "
Sep 10 14:47:47.693905 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.693871    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-csr-dns-cert\") pod \"7936579f-3755-475c-a158-9496c5226520\" (UID: \"7936579f-3755-475c-a158-9496c5226520\") "
Sep 10 14:47:47.694466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.694433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7936579f-3755-475c-a158-9496c5226520-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7936579f-3755-475c-a158-9496c5226520" (UID: "7936579f-3755-475c-a158-9496c5226520"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:47:47.697637 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.697361    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7936579f-3755-475c-a158-9496c5226520" (UID: "7936579f-3755-475c-a158-9496c5226520"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:47.697637 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.697478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7936579f-3755-475c-a158-9496c5226520-local-certs" (OuterVolumeSpecName: "local-certs") pod "7936579f-3755-475c-a158-9496c5226520" (UID: "7936579f-3755-475c-a158-9496c5226520"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:47:47.697637 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.697497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-istio-token" (OuterVolumeSpecName: "istio-token") pod "7936579f-3755-475c-a158-9496c5226520" (UID: "7936579f-3755-475c-a158-9496c5226520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:47.697888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.697750    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7936579f-3755-475c-a158-9496c5226520" (UID: "7936579f-3755-475c-a158-9496c5226520"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:47.697943 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.697900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-kube-api-access-2kmw9" (OuterVolumeSpecName: "kube-api-access-2kmw9") pod "7936579f-3755-475c-a158-9496c5226520" (UID: "7936579f-3755-475c-a158-9496c5226520"). InnerVolumeSpecName "kube-api-access-2kmw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:47:47.697998 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.697977    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-cacerts" (OuterVolumeSpecName: "cacerts") pod "7936579f-3755-475c-a158-9496c5226520" (UID: "7936579f-3755-475c-a158-9496c5226520"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:47:47.795158 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.795123    2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-kubeconfig\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:47:47.795158 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.795152    2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-cacerts\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:47:47.795158 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.795163    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-istio-token\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:47:47.795437 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.795171    2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7936579f-3755-475c-a158-9496c5226520-local-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:47:47.795437 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.795180    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kmw9\" (UniqueName: \"kubernetes.io/projected/7936579f-3755-475c-a158-9496c5226520-kube-api-access-2kmw9\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:47:47.795437 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.795188    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7936579f-3755-475c-a158-9496c5226520-istio-csr-dns-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:47:47.795437 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:47.795196    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7936579f-3755-475c-a158-9496c5226520-istio-csr-ca-configmap\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:47:48.501071 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:48.501029    2563 generic.go:356] "Generic (PLEG): container finished" podID="7936579f-3755-475c-a158-9496c5226520" containerID="22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5" exitCode=0
Sep 10 14:47:48.501460 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:48.501082    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb" event={"ID":"7936579f-3755-475c-a158-9496c5226520","Type":"ContainerDied","Data":"22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5"}
Sep 10 14:47:48.501460 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:48.501088    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"
Sep 10 14:47:48.501460 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:48.501129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb" event={"ID":"7936579f-3755-475c-a158-9496c5226520","Type":"ContainerDied","Data":"769ae26419a5fdc84f62ce876270a94760b9a8599bbca904e2e0668cde76a279"}
Sep 10 14:47:48.501460 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:48.501148    2563 scope.go:117] "RemoveContainer" containerID="22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5"
Sep 10 14:47:48.520088 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:48.520056    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"]
Sep 10 14:47:48.526555 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:48.526528    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-st2lb"]
Sep 10 14:47:49.011004 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:49.010971    2563 scope.go:117] "RemoveContainer" containerID="22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5"
Sep 10 14:47:49.011430 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:47:49.011394    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5\": container with ID starting with 22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5 not found: ID does not exist" containerID="22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5"
Sep 10 14:47:49.011545 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:49.011436    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5"} err="failed to get container status \"22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5\": rpc error: code = NotFound desc = could not find container \"22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5\": container with ID starting with 22b2b3107f4a187a53e9120823f4a6e920fbb1e1c7acc7f4593ee702a64136d5 not found: ID does not exist"
Sep 10 14:47:50.031876 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:50.031844    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7936579f-3755-475c-a158-9496c5226520" path="/var/lib/kubelet/pods/7936579f-3755-475c-a158-9496c5226520/volumes"
Sep 10 14:47:50.514670 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:50.514633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4" event={"ID":"f881eda7-2589-4f11-818b-12fbbdc66305","Type":"ContainerStarted","Data":"a786d008f2bfbf3358b60a9103271fb58f95b545bc10466910f091cbb76d4779"}
Sep 10 14:47:50.514869 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:50.514739    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:47:50.516341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:50.516316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j" event={"ID":"37acfe05-5214-4191-abb7-741d7d0e46b4","Type":"ContainerStarted","Data":"b032c1182616831b02f22e0fb5c39081457592fc619af472f5d0c3bd1cef7208"}
Sep 10 14:47:50.516441 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:50.516400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:47:50.530800 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:50.530755    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4" podStartSLOduration=1.770293864 podStartE2EDuration="5.530741714s" totalImagesPullingTime="3.76044785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:50.529844692 +0000 UTC m=+341.006904659" watchObservedRunningTime="2026-09-10 14:47:50.530741714 +0000 UTC m=+341.007801694"
Sep 10 14:47:50.546548 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:47:50.546493    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j" podStartSLOduration=2.953319277 podStartE2EDuration="5.546484163s" totalImagesPullingTime="2.593164886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:50.545090816 +0000 UTC m=+341.022150782" watchObservedRunningTime="2026-09-10 14:47:50.546484163 +0000 UTC m=+341.023544136"
Sep 10 14:48:21.523384 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:21.523351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-ltrn4"
Sep 10 14:48:21.526622 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:21.526599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:48:37.248305 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:37.248275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:48:37.387906 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:37.387867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:48:53.838616 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:53.838580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:48:53.919463 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:53.919436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:48:56.104989 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:56.104955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:48:56.219534 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:56.219474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:48:58.357699 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.357667    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"]
Sep 10 14:48:58.358056 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.357958    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7936579f-3755-475c-a158-9496c5226520" containerName="discovery"
Sep 10 14:48:58.358056 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.357970    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7936579f-3755-475c-a158-9496c5226520" containerName="discovery"
Sep 10 14:48:58.358056 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.358029    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7936579f-3755-475c-a158-9496c5226520" containerName="discovery"
Sep 10 14:48:58.361036 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.361019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"
Sep 10 14:48:58.392869 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.392845    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 14:48:58.393369 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.393351    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jhm2r\""
Sep 10 14:48:58.393654 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.393639    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 14:48:58.400165 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.400146    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"]
Sep 10 14:48:58.402661 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.402642    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 14:48:58.466484 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.466456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjkmr\" (UniqueName: \"kubernetes.io/projected/70d0014b-d871-4333-87f4-d8eec73976bc-kube-api-access-gjkmr\") pod \"dns-operator-controller-manager-7b84cd7c9c-gq5m4\" (UID: \"70d0014b-d871-4333-87f4-d8eec73976bc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"
Sep 10 14:48:58.567875 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.567845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjkmr\" (UniqueName: \"kubernetes.io/projected/70d0014b-d871-4333-87f4-d8eec73976bc-kube-api-access-gjkmr\") pod \"dns-operator-controller-manager-7b84cd7c9c-gq5m4\" (UID: \"70d0014b-d871-4333-87f4-d8eec73976bc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"
Sep 10 14:48:58.588931 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.588901    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjkmr\" (UniqueName: \"kubernetes.io/projected/70d0014b-d871-4333-87f4-d8eec73976bc-kube-api-access-gjkmr\") pod \"dns-operator-controller-manager-7b84cd7c9c-gq5m4\" (UID: \"70d0014b-d871-4333-87f4-d8eec73976bc\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"
Sep 10 14:48:58.670747 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.670708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"
Sep 10 14:48:58.991746 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:58.991669    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"]
Sep 10 14:48:58.999415 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:48:58.999383    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d0014b_d871_4333_87f4_d8eec73976bc.slice/crio-a7b66729c047e467ff8c23e5687dbbdfb2af6a54355466778576a68cd6e13a5b WatchSource:0}: Error finding container a7b66729c047e467ff8c23e5687dbbdfb2af6a54355466778576a68cd6e13a5b: Status 404 returned error can't find the container with id a7b66729c047e467ff8c23e5687dbbdfb2af6a54355466778576a68cd6e13a5b
Sep 10 14:48:59.213693 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:59.213657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:48:59.590458 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:59.590422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:48:59.803768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:59.803739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4" event={"ID":"70d0014b-d871-4333-87f4-d8eec73976bc","Type":"ContainerStarted","Data":"a7b66729c047e467ff8c23e5687dbbdfb2af6a54355466778576a68cd6e13a5b"}
Sep 10 14:48:59.941616 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:48:59.941557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:00.277446 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:00.277294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:00.444129 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:00.444097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:00.842331 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:00.842292    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:01.118219 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:01.118102    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:01.570618 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:01.570582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:01.815583 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:01.815551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4" event={"ID":"70d0014b-d871-4333-87f4-d8eec73976bc","Type":"ContainerStarted","Data":"81da8175269ee13aad597332f99f7c1278a25928d069fed2116b1568933b8ea9"}
Sep 10 14:49:01.815776 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:01.815667    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"
Sep 10 14:49:01.858693 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:01.858602    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4" podStartSLOduration=1.225312376 podStartE2EDuration="3.858585716s" totalImagesPullingTime="2.63327334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:01.857634675 +0000 UTC m=+412.334694635" watchObservedRunningTime="2026-09-10 14:49:01.858585716 +0000 UTC m=+412.335645660"
Sep 10 14:49:03.462296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.462267    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"]
Sep 10 14:49:03.465735 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.465719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:03.469616 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.469590    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wzp2t\""
Sep 10 14:49:03.498138 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.498114    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"]
Sep 10 14:49:03.512873 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.512846    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlckz\" (UniqueName: \"kubernetes.io/projected/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-kube-api-access-xlckz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" (UID: \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:03.513000 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.512877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" (UID: \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:03.591100 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.591067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:03.614143 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.614117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlckz\" (UniqueName: \"kubernetes.io/projected/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-kube-api-access-xlckz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" (UID: \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:03.614226 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.614152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" (UID: \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:03.614623 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.614606    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" (UID: \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:03.637065 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.637045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlckz\" (UniqueName: \"kubernetes.io/projected/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-kube-api-access-xlckz\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" (UID: \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:03.776311 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:03.776231    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:04.019935 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:04.019901    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:04.085986 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:04.085692    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"]
Sep 10 14:49:04.090868 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:49:04.090839    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3ec5bf_22bc_418f_b8e9_8da5fa95719b.slice/crio-295d57d794197986cb56087d7cd9c563da967ba158ef1fa27cb16bca20a12c36 WatchSource:0}: Error finding container 295d57d794197986cb56087d7cd9c563da967ba158ef1fa27cb16bca20a12c36: Status 404 returned error can't find the container with id 295d57d794197986cb56087d7cd9c563da967ba158ef1fa27cb16bca20a12c36
Sep 10 14:49:04.830618 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:04.830583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc" event={"ID":"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b","Type":"ContainerStarted","Data":"295d57d794197986cb56087d7cd9c563da967ba158ef1fa27cb16bca20a12c36"}
Sep 10 14:49:06.033646 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:06.033604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:06.797302 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:06.797260    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dvrf9"]
Sep 10 14:49:06.800663 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:06.800629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9"
Sep 10 14:49:06.803805 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:06.803750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:06.804358 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:06.804340    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-f28sq\""
Sep 10 14:49:06.840761 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:06.840727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhts\" (UniqueName: \"kubernetes.io/projected/5f5e9277-b986-4595-8cbf-93da02b83a60-kube-api-access-djhts\") pod \"authorino-operator-76b478bd78-dvrf9\" (UID: \"5f5e9277-b986-4595-8cbf-93da02b83a60\") " pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9"
Sep 10 14:49:06.860271 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:06.860241    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dvrf9"]
Sep 10 14:49:06.941292 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:06.941257    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhts\" (UniqueName: \"kubernetes.io/projected/5f5e9277-b986-4595-8cbf-93da02b83a60-kube-api-access-djhts\") pod \"authorino-operator-76b478bd78-dvrf9\" (UID: \"5f5e9277-b986-4595-8cbf-93da02b83a60\") " pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9"
Sep 10 14:49:06.982618 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:06.982586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhts\" (UniqueName: \"kubernetes.io/projected/5f5e9277-b986-4595-8cbf-93da02b83a60-kube-api-access-djhts\") pod \"authorino-operator-76b478bd78-dvrf9\" (UID: \"5f5e9277-b986-4595-8cbf-93da02b83a60\") " pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9"
Sep 10 14:49:07.115222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:07.115133    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9"
Sep 10 14:49:07.159832 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:07.159790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:08.433059 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.433028    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-dvrf9"]
Sep 10 14:49:08.437214 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:49:08.437184    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5e9277_b986_4595_8cbf_93da02b83a60.slice/crio-ba994a8434b2ba66f04b8daf9be249337cbd3244a5876d29c8787a0651b980d1 WatchSource:0}: Error finding container ba994a8434b2ba66f04b8daf9be249337cbd3244a5876d29c8787a0651b980d1: Status 404 returned error can't find the container with id ba994a8434b2ba66f04b8daf9be249337cbd3244a5876d29c8787a0651b980d1
Sep 10 14:49:08.842191 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.842096    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"]
Sep 10 14:49:08.845602 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.845586    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:08.849220 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.849192    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dk78x\""
Sep 10 14:49:08.854907 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.854869    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc" event={"ID":"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b","Type":"ContainerStarted","Data":"0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca"}
Sep 10 14:49:08.855056 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.854938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:08.856275 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.856256    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9" event={"ID":"5f5e9277-b986-4595-8cbf-93da02b83a60","Type":"ContainerStarted","Data":"ba994a8434b2ba66f04b8daf9be249337cbd3244a5876d29c8787a0651b980d1"}
Sep 10 14:49:08.874847 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.874821    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"]
Sep 10 14:49:08.948002 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.947953    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc" podStartSLOduration=1.837505671 podStartE2EDuration="5.947936908s" totalImagesPullingTime="4.110431237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:08.946178485 +0000 UTC m=+419.423238426" watchObservedRunningTime="2026-09-10 14:49:08.947936908 +0000 UTC m=+419.424996851"
Sep 10 14:49:08.959443 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:08.959413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrth\" (UniqueName: \"kubernetes.io/projected/485790ff-39bf-4900-8cee-f5a4f8eabab1-kube-api-access-ftrth\") pod \"limitador-operator-controller-manager-74495c69f8-4dqg8\" (UID: \"485790ff-39bf-4900-8cee-f5a4f8eabab1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:09.059873 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:09.059838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrth\" (UniqueName: \"kubernetes.io/projected/485790ff-39bf-4900-8cee-f5a4f8eabab1-kube-api-access-ftrth\") pod \"limitador-operator-controller-manager-74495c69f8-4dqg8\" (UID: \"485790ff-39bf-4900-8cee-f5a4f8eabab1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:09.068689 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:09.068662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrth\" (UniqueName: \"kubernetes.io/projected/485790ff-39bf-4900-8cee-f5a4f8eabab1-kube-api-access-ftrth\") pod \"limitador-operator-controller-manager-74495c69f8-4dqg8\" (UID: \"485790ff-39bf-4900-8cee-f5a4f8eabab1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:09.157010 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:09.156981    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:09.440819 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:09.440779    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"]
Sep 10 14:49:09.443413 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:49:09.443383    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485790ff_39bf_4900_8cee_f5a4f8eabab1.slice/crio-5ad2fccaa52188d22702a50417cd152dd67177309b4c644c09f4ee48b77dfb92 WatchSource:0}: Error finding container 5ad2fccaa52188d22702a50417cd152dd67177309b4c644c09f4ee48b77dfb92: Status 404 returned error can't find the container with id 5ad2fccaa52188d22702a50417cd152dd67177309b4c644c09f4ee48b77dfb92
Sep 10 14:49:09.741684 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:09.741602    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:09.862558 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:09.862494    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8" event={"ID":"485790ff-39bf-4900-8cee-f5a4f8eabab1","Type":"ContainerStarted","Data":"5ad2fccaa52188d22702a50417cd152dd67177309b4c644c09f4ee48b77dfb92"}
Sep 10 14:49:09.863431 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:09.863401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:10.868084 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:10.868042    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9" event={"ID":"5f5e9277-b986-4595-8cbf-93da02b83a60","Type":"ContainerStarted","Data":"8ce00e59d7a023cf3f3e356a3e122a5fe97ae526e1273c72609b5d26d61fba92"}
Sep 10 14:49:10.868489 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:10.868188    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9"
Sep 10 14:49:10.896123 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:10.896071    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9" podStartSLOduration=3.191633292 podStartE2EDuration="4.896053905s" totalImagesPullingTime="1.704420613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:10.895221732 +0000 UTC m=+421.372281686" watchObservedRunningTime="2026-09-10 14:49:10.896053905 +0000 UTC m=+421.373113848"
Sep 10 14:49:11.211756 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:11.211712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:11.364904 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:11.364862    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:11.463777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:11.463690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:11.567495 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:11.567464    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:11.684045 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:11.684010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:12.822881 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:12.822855    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-gq5m4"
Sep 10 14:49:14.837248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:14.837212    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:15.087330 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:15.087254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:15.191536 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:15.191487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:15.341728 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:15.341624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:15.441973 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:15.441929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:15.542298 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:15.542264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:17.900431 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:17.900403    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8" event={"ID":"485790ff-39bf-4900-8cee-f5a4f8eabab1","Type":"ContainerStarted","Data":"779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3"}
Sep 10 14:49:17.900772 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:17.900460    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:17.929419 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:17.929368    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8" podStartSLOduration=1.542479027 podStartE2EDuration="9.929353115s" totalImagesPullingTime="8.386874088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:17.928788103 +0000 UTC m=+428.405848047" watchObservedRunningTime="2026-09-10 14:49:17.929353115 +0000 UTC m=+428.406413058"
Sep 10 14:49:19.865263 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:19.865231    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:21.874795 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:21.874762    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-dvrf9"
Sep 10 14:49:22.442259 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.442228    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"]
Sep 10 14:49:22.442435 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.442416    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" containerName="manager" containerID="cri-o://0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca" gracePeriod=2
Sep 10 14:49:22.465847 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.465821    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"]
Sep 10 14:49:22.466989 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.466963    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.492806 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.492779    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"]
Sep 10 14:49:22.492955 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.492937    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" containerName="manager" containerID="cri-o://779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3" gracePeriod=2
Sep 10 14:49:22.494795 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.494776    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.499458 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.499424    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.527657 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.527635    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"]
Sep 10 14:49:22.532362 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.532329    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.535343 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.535308    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.555272 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.555249    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"]
Sep 10 14:49:22.555584 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.555571    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" containerName="manager"
Sep 10 14:49:22.555639 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.555587    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" containerName="manager"
Sep 10 14:49:22.555639 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.555606    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" containerName="manager"
Sep 10 14:49:22.555639 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.555611    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" containerName="manager"
Sep 10 14:49:22.555726 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.555667    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" containerName="manager"
Sep 10 14:49:22.555726 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.555677    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" containerName="manager"
Sep 10 14:49:22.558392 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.558377    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:22.561804 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.561766    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.574689 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.574662    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.615317 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.615293    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"]
Sep 10 14:49:22.679405 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.679333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/226ed742-0ed4-430a-b4fb-6a3cf85cafee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v5dtb\" (UID: \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:22.679405 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.679380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd6gq\" (UniqueName: \"kubernetes.io/projected/226ed742-0ed4-430a-b4fb-6a3cf85cafee-kube-api-access-wd6gq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v5dtb\" (UID: \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:22.726426 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.726396    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"]
Sep 10 14:49:22.729732 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.729713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"
Sep 10 14:49:22.737873 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.737851    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.740895 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.740873    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.776262 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.776236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"]
Sep 10 14:49:22.780005 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.779968    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/226ed742-0ed4-430a-b4fb-6a3cf85cafee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v5dtb\" (UID: \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:22.780137 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.780019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd6gq\" (UniqueName: \"kubernetes.io/projected/226ed742-0ed4-430a-b4fb-6a3cf85cafee-kube-api-access-wd6gq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v5dtb\" (UID: \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:22.780365 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.780348    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/226ed742-0ed4-430a-b4fb-6a3cf85cafee-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v5dtb\" (UID: \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:22.824494 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.824471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd6gq\" (UniqueName: \"kubernetes.io/projected/226ed742-0ed4-430a-b4fb-6a3cf85cafee-kube-api-access-wd6gq\") pod \"kuadrant-operator-controller-manager-76f8f686bf-v5dtb\" (UID: \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:22.859621 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.859583    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.867840 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.867811    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.869657 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.869637    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.873048 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.873031    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.879257 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.879236    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:22.880421 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.880404    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftrth\" (UniqueName: \"kubernetes.io/projected/485790ff-39bf-4900-8cee-f5a4f8eabab1-kube-api-access-ftrth\") pod \"485790ff-39bf-4900-8cee-f5a4f8eabab1\" (UID: \"485790ff-39bf-4900-8cee-f5a4f8eabab1\") "
Sep 10 14:49:22.880497 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.880486    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlckz\" (UniqueName: \"kubernetes.io/projected/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-kube-api-access-xlckz\") pod \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\" (UID: \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\") "
Sep 10 14:49:22.880579 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.880542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-extensions-socket-volume\") pod \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\" (UID: \"8d3ec5bf-22bc-418f-b8e9-8da5fa95719b\") "
Sep 10 14:49:22.880733 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.880714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdx4x\" (UniqueName: \"kubernetes.io/projected/372a2ece-1e88-4404-9651-5f7975f98587-kube-api-access-cdx4x\") pod \"limitador-operator-controller-manager-74495c69f8-s86pg\" (UID: \"372a2ece-1e88-4404-9651-5f7975f98587\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"
Sep 10 14:49:22.880941 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.880922    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" (UID: "8d3ec5bf-22bc-418f-b8e9-8da5fa95719b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:22.882664 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.882636    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/485790ff-39bf-4900-8cee-f5a4f8eabab1-kube-api-access-ftrth" (OuterVolumeSpecName: "kube-api-access-ftrth") pod "485790ff-39bf-4900-8cee-f5a4f8eabab1" (UID: "485790ff-39bf-4900-8cee-f5a4f8eabab1"). InnerVolumeSpecName "kube-api-access-ftrth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:22.882747 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.882691    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-kube-api-access-xlckz" (OuterVolumeSpecName: "kube-api-access-xlckz") pod "8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" (UID: "8d3ec5bf-22bc-418f-b8e9-8da5fa95719b"). InnerVolumeSpecName "kube-api-access-xlckz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:22.925623 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.925590    2563 generic.go:356] "Generic (PLEG): container finished" podID="485790ff-39bf-4900-8cee-f5a4f8eabab1" containerID="779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3" exitCode=0
Sep 10 14:49:22.925817 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.925646    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.925817 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.925702    2563 scope.go:117] "RemoveContainer" containerID="779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3"
Sep 10 14:49:22.928147 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.928104    2563 generic.go:356] "Generic (PLEG): container finished" podID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" containerID="0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca" exitCode=0
Sep 10 14:49:22.928274 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.928215    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.928999 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.928685    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.931471 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.931289    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.933539 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.933406    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.940384 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.940360    2563 scope.go:117] "RemoveContainer" containerID="779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3"
Sep 10 14:49:22.941272 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.940754    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.943142 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.943111    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3\": container with ID starting with 779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3 not found: ID does not exist" containerID="779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3"
Sep 10 14:49:22.943233 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.943155    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3"} err="failed to get container status \"779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3\": rpc error: code = NotFound desc = could not find container \"779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3\": container with ID starting with 779eaa1dc5bae345bee585d141efff785edc3b8b46c7db6f68144bd892ab6ac3 not found: ID does not exist"
Sep 10 14:49:22.943233 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.943193    2563 scope.go:117] "RemoveContainer" containerID="0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca"
Sep 10 14:49:22.952387 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.952370    2563 scope.go:117] "RemoveContainer" containerID="0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca"
Sep 10 14:49:22.952682 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.952663    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca\": container with ID starting with 0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca not found: ID does not exist" containerID="0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca"
Sep 10 14:49:22.952747 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.952691    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca"} err="failed to get container status \"0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca\": rpc error: code = NotFound desc = could not find container \"0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca\": container with ID starting with 0ea152cabeb0e29334a659d11f846fa94c0ecec204200f0544b05b4e4577d2ca not found: ID does not exist"
Sep 10 14:49:22.967679 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.967626    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:22.981992 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.981974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdx4x\" (UniqueName: \"kubernetes.io/projected/372a2ece-1e88-4404-9651-5f7975f98587-kube-api-access-cdx4x\") pod \"limitador-operator-controller-manager-74495c69f8-s86pg\" (UID: \"372a2ece-1e88-4404-9651-5f7975f98587\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"
Sep 10 14:49:22.982076 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.982064    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlckz\" (UniqueName: \"kubernetes.io/projected/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-kube-api-access-xlckz\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:49:22.982076 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.982074    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b-extensions-socket-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:49:22.982141 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:22.982084    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftrth\" (UniqueName: \"kubernetes.io/projected/485790ff-39bf-4900-8cee-f5a4f8eabab1-kube-api-access-ftrth\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:49:22.986215 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.986192    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.988369 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.988348    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:22.999074 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:22.999051    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:23.055054 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.055024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdx4x\" (UniqueName: \"kubernetes.io/projected/372a2ece-1e88-4404-9651-5f7975f98587-kube-api-access-cdx4x\") pod \"limitador-operator-controller-manager-74495c69f8-s86pg\" (UID: \"372a2ece-1e88-4404-9651-5f7975f98587\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"
Sep 10 14:49:23.205470 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.205442    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"]
Sep 10 14:49:23.205607 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:49:23.205588    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226ed742_0ed4_430a_b4fb_6a3cf85cafee.slice/crio-e2cbfaf68db3398aa69c710e315d94a7d030e836c1c8946490c36b19717a045b WatchSource:0}: Error finding container e2cbfaf68db3398aa69c710e315d94a7d030e836c1c8946490c36b19717a045b: Status 404 returned error can't find the container with id e2cbfaf68db3398aa69c710e315d94a7d030e836c1c8946490c36b19717a045b
Sep 10 14:49:23.347070 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.347035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"
Sep 10 14:49:23.518244 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.518153    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:23.668479 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.668454    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"]
Sep 10 14:49:23.670767 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:49:23.670738    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372a2ece_1e88_4404_9651_5f7975f98587.slice/crio-537e08dfdd3145ed4242c6cd80dff43ce37216223a85e2283d9af67decc1721b WatchSource:0}: Error finding container 537e08dfdd3145ed4242c6cd80dff43ce37216223a85e2283d9af67decc1721b: Status 404 returned error can't find the container with id 537e08dfdd3145ed4242c6cd80dff43ce37216223a85e2283d9af67decc1721b
Sep 10 14:49:23.705466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.705438    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:23.935238 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.935207    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb" event={"ID":"226ed742-0ed4-430a-b4fb-6a3cf85cafee","Type":"ContainerStarted","Data":"2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18"}
Sep 10 14:49:23.935675 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.935256    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb" event={"ID":"226ed742-0ed4-430a-b4fb-6a3cf85cafee","Type":"ContainerStarted","Data":"e2cbfaf68db3398aa69c710e315d94a7d030e836c1c8946490c36b19717a045b"}
Sep 10 14:49:23.935675 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.935320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:23.938281 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.938255    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg" event={"ID":"372a2ece-1e88-4404-9651-5f7975f98587","Type":"ContainerStarted","Data":"1ef89c095c835da45c16b3f087e361df24e1a38730789bf1ca95a42e1c02bd73"}
Sep 10 14:49:23.938419 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.938302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg" event={"ID":"372a2ece-1e88-4404-9651-5f7975f98587","Type":"ContainerStarted","Data":"537e08dfdd3145ed4242c6cd80dff43ce37216223a85e2283d9af67decc1721b"}
Sep 10 14:49:23.938419 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:23.938328    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"
Sep 10 14:49:23.941269 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:23.941248    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:23.949104 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:23.949085    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:24.003049 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:24.003014    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rr9cc\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rr9cc"
Sep 10 14:49:24.003570 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:24.003535    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb" podStartSLOduration=2.003521954 podStartE2EDuration="2.003521954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:24.001373674 +0000 UTC m=+434.478433629" watchObservedRunningTime="2026-09-10 14:49:24.003521954 +0000 UTC m=+434.480581894"
Sep 10 14:49:24.031879 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:24.031853    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" path="/var/lib/kubelet/pods/485790ff-39bf-4900-8cee-f5a4f8eabab1/volumes"
Sep 10 14:49:24.032188 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:24.032176    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d3ec5bf-22bc-418f-b8e9-8da5fa95719b" path="/var/lib/kubelet/pods/8d3ec5bf-22bc-418f-b8e9-8da5fa95719b/volumes"
Sep 10 14:49:24.049646 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:24.049603    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg" podStartSLOduration=2.0495905150000002 podStartE2EDuration="2.049590515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:24.048244044 +0000 UTC m=+434.525303988" watchObservedRunningTime="2026-09-10 14:49:24.049590515 +0000 UTC m=+434.526650458"
Sep 10 14:49:24.053795 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:24.053773    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-4dqg8\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="485790ff-39bf-4900-8cee-f5a4f8eabab1" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-4dqg8"
Sep 10 14:49:26.450750 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.450716    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:26.708696 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.708612    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:26.739851 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.739823    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"]
Sep 10 14:49:26.743535 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.743497    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:26.812541 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.812486    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkhzq\" (UniqueName: \"kubernetes.io/projected/4052641e-da82-44a8-9f3f-6459c0c324c7-kube-api-access-jkhzq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-82wbn\" (UID: \"4052641e-da82-44a8-9f3f-6459c0c324c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:26.812710 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.812603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4052641e-da82-44a8-9f3f-6459c0c324c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-82wbn\" (UID: \"4052641e-da82-44a8-9f3f-6459c0c324c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:26.877687 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.877657    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"]
Sep 10 14:49:26.913952 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.913921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4052641e-da82-44a8-9f3f-6459c0c324c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-82wbn\" (UID: \"4052641e-da82-44a8-9f3f-6459c0c324c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:26.914120 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.913992    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkhzq\" (UniqueName: \"kubernetes.io/projected/4052641e-da82-44a8-9f3f-6459c0c324c7-kube-api-access-jkhzq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-82wbn\" (UID: \"4052641e-da82-44a8-9f3f-6459c0c324c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:26.914312 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.914293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4052641e-da82-44a8-9f3f-6459c0c324c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-82wbn\" (UID: \"4052641e-da82-44a8-9f3f-6459c0c324c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:26.935048 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:26.935022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkhzq\" (UniqueName: \"kubernetes.io/projected/4052641e-da82-44a8-9f3f-6459c0c324c7-kube-api-access-jkhzq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-82wbn\" (UID: \"4052641e-da82-44a8-9f3f-6459c0c324c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:27.052722 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:27.052632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:27.353875 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:27.353416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"]
Sep 10 14:49:27.358342 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:49:27.358318    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4052641e_da82_44a8_9f3f_6459c0c324c7.slice/crio-5a96864119c1099a92cf6cb39cc19731705c2c04aa9008e323c72c08d928e860 WatchSource:0}: Error finding container 5a96864119c1099a92cf6cb39cc19731705c2c04aa9008e323c72c08d928e860: Status 404 returned error can't find the container with id 5a96864119c1099a92cf6cb39cc19731705c2c04aa9008e323c72c08d928e860
Sep 10 14:49:27.958132 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:27.958097    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn" event={"ID":"4052641e-da82-44a8-9f3f-6459c0c324c7","Type":"ContainerStarted","Data":"0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a"}
Sep 10 14:49:27.958639 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:27.958151    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:27.958639 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:27.958168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn" event={"ID":"4052641e-da82-44a8-9f3f-6459c0c324c7","Type":"ContainerStarted","Data":"5a96864119c1099a92cf6cb39cc19731705c2c04aa9008e323c72c08d928e860"}
Sep 10 14:49:28.010926 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:28.010871    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn" podStartSLOduration=2.010857022 podStartE2EDuration="2.010857022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:49:28.005717581 +0000 UTC m=+438.482777548" watchObservedRunningTime="2026-09-10 14:49:28.010857022 +0000 UTC m=+438.487916965"
Sep 10 14:49:29.258290 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:29.258256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:29.462834 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:29.462801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:34.132394 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:34.132310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:34.232677 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:34.232645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:34.330663 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:34.330630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:34.491635 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:34.491601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:34.586956 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:34.586920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:34.687906 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:34.687875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:34.945983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:34.945950    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-s86pg"
Sep 10 14:49:34.946152 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:34.946020    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:38.965009 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:38.964975    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 14:49:39.045181 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:39.045145    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"]
Sep 10 14:49:39.045402 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:39.045379    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb" podUID="226ed742-0ed4-430a-b4fb-6a3cf85cafee" containerName="manager" containerID="cri-o://2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18" gracePeriod=10
Sep 10 14:49:39.291062 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:39.291034    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:39.419329 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:39.419301    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd6gq\" (UniqueName: \"kubernetes.io/projected/226ed742-0ed4-430a-b4fb-6a3cf85cafee-kube-api-access-wd6gq\") pod \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\" (UID: \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\") "
Sep 10 14:49:39.419539 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:39.419395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/226ed742-0ed4-430a-b4fb-6a3cf85cafee-extensions-socket-volume\") pod \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\" (UID: \"226ed742-0ed4-430a-b4fb-6a3cf85cafee\") "
Sep 10 14:49:39.419800 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:39.419777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226ed742-0ed4-430a-b4fb-6a3cf85cafee-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "226ed742-0ed4-430a-b4fb-6a3cf85cafee" (UID: "226ed742-0ed4-430a-b4fb-6a3cf85cafee"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:49:39.421309 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:39.421286    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226ed742-0ed4-430a-b4fb-6a3cf85cafee-kube-api-access-wd6gq" (OuterVolumeSpecName: "kube-api-access-wd6gq") pod "226ed742-0ed4-430a-b4fb-6a3cf85cafee" (UID: "226ed742-0ed4-430a-b4fb-6a3cf85cafee"). InnerVolumeSpecName "kube-api-access-wd6gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:49:39.520921 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:39.520881    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/226ed742-0ed4-430a-b4fb-6a3cf85cafee-extensions-socket-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:49:39.520921 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:39.520917    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd6gq\" (UniqueName: \"kubernetes.io/projected/226ed742-0ed4-430a-b4fb-6a3cf85cafee-kube-api-access-wd6gq\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:49:40.013807 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:40.013769    2563 generic.go:356] "Generic (PLEG): container finished" podID="226ed742-0ed4-430a-b4fb-6a3cf85cafee" containerID="2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18" exitCode=0
Sep 10 14:49:40.014181 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:40.013813    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"
Sep 10 14:49:40.014181 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:40.013855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb" event={"ID":"226ed742-0ed4-430a-b4fb-6a3cf85cafee","Type":"ContainerDied","Data":"2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18"}
Sep 10 14:49:40.014181 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:40.013911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb" event={"ID":"226ed742-0ed4-430a-b4fb-6a3cf85cafee","Type":"ContainerDied","Data":"e2cbfaf68db3398aa69c710e315d94a7d030e836c1c8946490c36b19717a045b"}
Sep 10 14:49:40.014181 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:40.013935    2563 scope.go:117] "RemoveContainer" containerID="2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18"
Sep 10 14:49:40.024624 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:40.024581    2563 scope.go:117] "RemoveContainer" containerID="2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18"
Sep 10 14:49:40.024895 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:49:40.024875    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18\": container with ID starting with 2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18 not found: ID does not exist" containerID="2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18"
Sep 10 14:49:40.024961 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:40.024903    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18"} err="failed to get container status \"2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18\": rpc error: code = NotFound desc = could not find container \"2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18\": container with ID starting with 2707c73d57c0cfd08c0a960128c7a80e7e936c03623f27c3d5e2545c5da0af18 not found: ID does not exist"
Sep 10 14:49:40.041730 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:40.041707    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"]
Sep 10 14:49:40.048572 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:40.048551    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-v5dtb"]
Sep 10 14:49:42.031125 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:42.031088    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="226ed742-0ed4-430a-b4fb-6a3cf85cafee" path="/var/lib/kubelet/pods/226ed742-0ed4-430a-b4fb-6a3cf85cafee/volumes"
Sep 10 14:49:42.932066 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:42.932027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:43.030969 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:43.030930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:45.729734 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:45.729703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:45.937970 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:45.937928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:46.039011 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:46.038910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:55.377806 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:55.377770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:55.533334 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:55.533302    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:58.004880 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.004846    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bcf77"]
Sep 10 14:49:58.005342 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.005162    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="226ed742-0ed4-430a-b4fb-6a3cf85cafee" containerName="manager"
Sep 10 14:49:58.005342 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.005174    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="226ed742-0ed4-430a-b4fb-6a3cf85cafee" containerName="manager"
Sep 10 14:49:58.005342 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.005234    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="226ed742-0ed4-430a-b4fb-6a3cf85cafee" containerName="manager"
Sep 10 14:49:58.009382 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.009364    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:49:58.011457 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.011434    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 14:49:58.011564 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.011460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7tmbm\""
Sep 10 14:49:58.017556 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.017536    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bcf77"]
Sep 10 14:49:58.067681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.067636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bd2c7433-2127-4168-a79c-4bc9bb0fa077-config-file\") pod \"limitador-limitador-8588749977-bcf77\" (UID: \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\") " pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:49:58.067830 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.067727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstl6\" (UniqueName: \"kubernetes.io/projected/bd2c7433-2127-4168-a79c-4bc9bb0fa077-kube-api-access-wstl6\") pod \"limitador-limitador-8588749977-bcf77\" (UID: \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\") " pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:49:58.088608 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.088576    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bcf77"]
Sep 10 14:49:58.098546 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.098503    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:58.157597 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.157547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:58.168593 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.168563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bd2c7433-2127-4168-a79c-4bc9bb0fa077-config-file\") pod \"limitador-limitador-8588749977-bcf77\" (UID: \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\") " pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:49:58.168697 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.168622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstl6\" (UniqueName: \"kubernetes.io/projected/bd2c7433-2127-4168-a79c-4bc9bb0fa077-kube-api-access-wstl6\") pod \"limitador-limitador-8588749977-bcf77\" (UID: \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\") " pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:49:58.169206 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.169188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bd2c7433-2127-4168-a79c-4bc9bb0fa077-config-file\") pod \"limitador-limitador-8588749977-bcf77\" (UID: \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\") " pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:49:58.178050 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.178025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstl6\" (UniqueName: \"kubernetes.io/projected/bd2c7433-2127-4168-a79c-4bc9bb0fa077-kube-api-access-wstl6\") pod \"limitador-limitador-8588749977-bcf77\" (UID: \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\") " pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:49:58.320991 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.320900    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:49:58.593500 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.593416    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bcf77"]
Sep 10 14:49:58.596411 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:49:58.596383    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2c7433_2127_4168_a79c_4bc9bb0fa077.slice/crio-50454ba4e425b525c9d4f95d06040dfb17954932e72592329ed7f97da810fba4 WatchSource:0}: Error finding container 50454ba4e425b525c9d4f95d06040dfb17954932e72592329ed7f97da810fba4: Status 404 returned error can't find the container with id 50454ba4e425b525c9d4f95d06040dfb17954932e72592329ed7f97da810fba4
Sep 10 14:49:58.820400 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.820369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:58.947621 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.947592    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-8m7zn"]
Sep 10 14:49:58.951983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.951965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8m7zn"
Sep 10 14:49:58.954856 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.954833    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w5vlc\""
Sep 10 14:49:58.956889 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.956866    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:49:58.967034 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:58.967011    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8m7zn"]
Sep 10 14:49:59.075751 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:59.075701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hfqc\" (UniqueName: \"kubernetes.io/projected/d1ce94f8-7a11-4614-ae5a-63175864be61-kube-api-access-4hfqc\") pod \"authorino-59c974bd48-8m7zn\" (UID: \"d1ce94f8-7a11-4614-ae5a-63175864be61\") " pod="kuadrant-system/authorino-59c974bd48-8m7zn"
Sep 10 14:49:59.099925 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:59.099895    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-bcf77" event={"ID":"bd2c7433-2127-4168-a79c-4bc9bb0fa077","Type":"ContainerStarted","Data":"50454ba4e425b525c9d4f95d06040dfb17954932e72592329ed7f97da810fba4"}
Sep 10 14:49:59.176961 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:59.176928    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hfqc\" (UniqueName: \"kubernetes.io/projected/d1ce94f8-7a11-4614-ae5a-63175864be61-kube-api-access-4hfqc\") pod \"authorino-59c974bd48-8m7zn\" (UID: \"d1ce94f8-7a11-4614-ae5a-63175864be61\") " pod="kuadrant-system/authorino-59c974bd48-8m7zn"
Sep 10 14:49:59.185085 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:59.185063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hfqc\" (UniqueName: \"kubernetes.io/projected/d1ce94f8-7a11-4614-ae5a-63175864be61-kube-api-access-4hfqc\") pod \"authorino-59c974bd48-8m7zn\" (UID: \"d1ce94f8-7a11-4614-ae5a-63175864be61\") " pod="kuadrant-system/authorino-59c974bd48-8m7zn"
Sep 10 14:49:59.262207 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:59.262130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8m7zn"
Sep 10 14:49:59.532206 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:49:59.532180    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8m7zn"]
Sep 10 14:49:59.534285 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:49:59.534248    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1ce94f8_7a11_4614_ae5a_63175864be61.slice/crio-fcd8b132d96e33d296a54e7885402601e22e3018f96086caaf8983ab77c5bb08 WatchSource:0}: Error finding container fcd8b132d96e33d296a54e7885402601e22e3018f96086caaf8983ab77c5bb08: Status 404 returned error can't find the container with id fcd8b132d96e33d296a54e7885402601e22e3018f96086caaf8983ab77c5bb08
Sep 10 14:50:00.106085 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:00.106047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-bcf77" event={"ID":"bd2c7433-2127-4168-a79c-4bc9bb0fa077","Type":"ContainerStarted","Data":"a6d0ec1f6652ae499872e479bf382e95c2b45f3f34a80f9346ac7ca1dac30819"}
Sep 10 14:50:00.106532 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:00.106202    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:50:00.107299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:00.107274    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8m7zn" event={"ID":"d1ce94f8-7a11-4614-ae5a-63175864be61","Type":"ContainerStarted","Data":"fcd8b132d96e33d296a54e7885402601e22e3018f96086caaf8983ab77c5bb08"}
Sep 10 14:50:00.123182 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:00.123136    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-bcf77" podStartSLOduration=1.808747918 podStartE2EDuration="3.123120606s" totalImagesPullingTime="1.314372688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:00.1215351 +0000 UTC m=+470.598595044" watchObservedRunningTime="2026-09-10 14:50:00.123120606 +0000 UTC m=+470.600180932"
Sep 10 14:50:02.121119 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:02.121078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8m7zn" event={"ID":"d1ce94f8-7a11-4614-ae5a-63175864be61","Type":"ContainerStarted","Data":"9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f"}
Sep 10 14:50:02.135902 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:02.135852    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-8m7zn" podStartSLOduration=2.181148088 podStartE2EDuration="4.135836965s" totalImagesPullingTime="1.954688877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:02.135213738 +0000 UTC m=+472.612273682" watchObservedRunningTime="2026-09-10 14:50:02.135836965 +0000 UTC m=+472.612896909"
Sep 10 14:50:11.115686 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:11.115650    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:50:12.444668 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:12.444639    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bcf77"]
Sep 10 14:50:12.445047 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:12.444834    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-bcf77" podUID="bd2c7433-2127-4168-a79c-4bc9bb0fa077" containerName="limitador" containerID="cri-o://a6d0ec1f6652ae499872e479bf382e95c2b45f3f34a80f9346ac7ca1dac30819" gracePeriod=30
Sep 10 14:50:13.175948 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:13.175915    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd2c7433-2127-4168-a79c-4bc9bb0fa077" containerID="a6d0ec1f6652ae499872e479bf382e95c2b45f3f34a80f9346ac7ca1dac30819" exitCode=0
Sep 10 14:50:13.176114 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:13.175997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-bcf77" event={"ID":"bd2c7433-2127-4168-a79c-4bc9bb0fa077","Type":"ContainerDied","Data":"a6d0ec1f6652ae499872e479bf382e95c2b45f3f34a80f9346ac7ca1dac30819"}
Sep 10 14:50:13.389826 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:13.389803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:50:13.500316 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:13.500246    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bd2c7433-2127-4168-a79c-4bc9bb0fa077-config-file\") pod \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\" (UID: \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\") "
Sep 10 14:50:13.500316 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:13.500293    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wstl6\" (UniqueName: \"kubernetes.io/projected/bd2c7433-2127-4168-a79c-4bc9bb0fa077-kube-api-access-wstl6\") pod \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\" (UID: \"bd2c7433-2127-4168-a79c-4bc9bb0fa077\") "
Sep 10 14:50:13.500697 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:13.500601    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2c7433-2127-4168-a79c-4bc9bb0fa077-config-file" (OuterVolumeSpecName: "config-file") pod "bd2c7433-2127-4168-a79c-4bc9bb0fa077" (UID: "bd2c7433-2127-4168-a79c-4bc9bb0fa077"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:50:13.502341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:13.502320    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2c7433-2127-4168-a79c-4bc9bb0fa077-kube-api-access-wstl6" (OuterVolumeSpecName: "kube-api-access-wstl6") pod "bd2c7433-2127-4168-a79c-4bc9bb0fa077" (UID: "bd2c7433-2127-4168-a79c-4bc9bb0fa077"). InnerVolumeSpecName "kube-api-access-wstl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:50:13.601466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:13.601439    2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bd2c7433-2127-4168-a79c-4bc9bb0fa077-config-file\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:50:13.601466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:13.601464    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstl6\" (UniqueName: \"kubernetes.io/projected/bd2c7433-2127-4168-a79c-4bc9bb0fa077-kube-api-access-wstl6\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:50:14.023741 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.023703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:14.181839 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.181799    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-bcf77"
Sep 10 14:50:14.181839 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.181831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-bcf77" event={"ID":"bd2c7433-2127-4168-a79c-4bc9bb0fa077","Type":"ContainerDied","Data":"50454ba4e425b525c9d4f95d06040dfb17954932e72592329ed7f97da810fba4"}
Sep 10 14:50:14.182082 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.181889    2563 scope.go:117] "RemoveContainer" containerID="a6d0ec1f6652ae499872e479bf382e95c2b45f3f34a80f9346ac7ca1dac30819"
Sep 10 14:50:14.208029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.208002    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bcf77"]
Sep 10 14:50:14.214426 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.214401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:14.222693 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.222668    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-bcf77"]
Sep 10 14:50:14.472996 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.472965    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"]
Sep 10 14:50:14.473312 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.473300    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd2c7433-2127-4168-a79c-4bc9bb0fa077" containerName="limitador"
Sep 10 14:50:14.473359 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.473314    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd2c7433-2127-4168-a79c-4bc9bb0fa077" containerName="limitador"
Sep 10 14:50:14.473393 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.473364    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd2c7433-2127-4168-a79c-4bc9bb0fa077" containerName="limitador"
Sep 10 14:50:14.477683 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.477667    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:14.480133 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.480112    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 14:50:14.480764 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.480744    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 14:50:14.480914 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.480779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 14:50:14.481025 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.480825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-c28kf\""
Sep 10 14:50:14.488611 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.488589    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"]
Sep 10 14:50:14.526021 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.525993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:14.610159 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.610131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh8qg\" (UniqueName: \"kubernetes.io/projected/0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7-kube-api-access-hh8qg\") pod \"postgres-69fb4594cf-kz8h5\" (UID: \"0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:14.610159 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.610167    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7-data\") pod \"postgres-69fb4594cf-kz8h5\" (UID: \"0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:14.711438 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.711390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7-data\") pod \"postgres-69fb4594cf-kz8h5\" (UID: \"0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:14.711606 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.711486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh8qg\" (UniqueName: \"kubernetes.io/projected/0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7-kube-api-access-hh8qg\") pod \"postgres-69fb4594cf-kz8h5\" (UID: \"0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:14.711803 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.711785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7-data\") pod \"postgres-69fb4594cf-kz8h5\" (UID: \"0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:14.721054 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.721023    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh8qg\" (UniqueName: \"kubernetes.io/projected/0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7-kube-api-access-hh8qg\") pod \"postgres-69fb4594cf-kz8h5\" (UID: \"0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:14.789084 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:14.789006    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:15.059839 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:15.059761    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"]
Sep 10 14:50:15.063081 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:50:15.063050    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1efb48_3e0d_4621_a8ca_3eddf45cfdf7.slice/crio-311b5ec8b74a52482cede9605b8cf1ed65e1d51dd61c03f45706d45af92f0520 WatchSource:0}: Error finding container 311b5ec8b74a52482cede9605b8cf1ed65e1d51dd61c03f45706d45af92f0520: Status 404 returned error can't find the container with id 311b5ec8b74a52482cede9605b8cf1ed65e1d51dd61c03f45706d45af92f0520
Sep 10 14:50:15.187975 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:15.187941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5" event={"ID":"0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7","Type":"ContainerStarted","Data":"311b5ec8b74a52482cede9605b8cf1ed65e1d51dd61c03f45706d45af92f0520"}
Sep 10 14:50:16.036316 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:16.036277    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd2c7433-2127-4168-a79c-4bc9bb0fa077" path="/var/lib/kubelet/pods/bd2c7433-2127-4168-a79c-4bc9bb0fa077/volumes"
Sep 10 14:50:20.219325 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:20.219290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5" event={"ID":"0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7","Type":"ContainerStarted","Data":"fe01110669c0b86bf095f9f0c7a792338c96848a08f8e58c445a2e160b6978f3"}
Sep 10 14:50:20.219863 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:20.219351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:20.234470 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:20.234411    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5" podStartSLOduration=1.476520802 podStartE2EDuration="6.234389154s" totalImagesPullingTime="4.757868352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:20.233168959 +0000 UTC m=+490.710228904" watchObservedRunningTime="2026-09-10 14:50:20.234389154 +0000 UTC m=+490.711449099"
Sep 10 14:50:26.254200 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:26.254172    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kz8h5"
Sep 10 14:50:26.585030 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:26.584937    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:26.684755 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:26.684720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:27.067586 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.067552    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-755c66cc4d-lv4ks"]
Sep 10 14:50:27.073450 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.073426    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:50:27.075405 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.075384    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 14:50:27.078588 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.078569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-755c66cc4d-lv4ks"]
Sep 10 14:50:27.230744 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.230711    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94f5ae91-8c00-49a1-8963-2bbe94157b60-tls-cert\") pod \"authorino-755c66cc4d-lv4ks\" (UID: \"94f5ae91-8c00-49a1-8963-2bbe94157b60\") " pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:50:27.230744 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.230753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzr8z\" (UniqueName: \"kubernetes.io/projected/94f5ae91-8c00-49a1-8963-2bbe94157b60-kube-api-access-gzr8z\") pod \"authorino-755c66cc4d-lv4ks\" (UID: \"94f5ae91-8c00-49a1-8963-2bbe94157b60\") " pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:50:27.332168 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.332092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94f5ae91-8c00-49a1-8963-2bbe94157b60-tls-cert\") pod \"authorino-755c66cc4d-lv4ks\" (UID: \"94f5ae91-8c00-49a1-8963-2bbe94157b60\") " pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:50:27.332168 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.332127    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzr8z\" (UniqueName: \"kubernetes.io/projected/94f5ae91-8c00-49a1-8963-2bbe94157b60-kube-api-access-gzr8z\") pod \"authorino-755c66cc4d-lv4ks\" (UID: \"94f5ae91-8c00-49a1-8963-2bbe94157b60\") " pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:50:27.334476 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.334444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94f5ae91-8c00-49a1-8963-2bbe94157b60-tls-cert\") pod \"authorino-755c66cc4d-lv4ks\" (UID: \"94f5ae91-8c00-49a1-8963-2bbe94157b60\") " pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:50:27.340296 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.340261    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzr8z\" (UniqueName: \"kubernetes.io/projected/94f5ae91-8c00-49a1-8963-2bbe94157b60-kube-api-access-gzr8z\") pod \"authorino-755c66cc4d-lv4ks\" (UID: \"94f5ae91-8c00-49a1-8963-2bbe94157b60\") " pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:50:27.384169 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.384146    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:50:27.657870 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:27.657795    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-755c66cc4d-lv4ks"]
Sep 10 14:50:27.660971 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:50:27.660945    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94f5ae91_8c00_49a1_8963_2bbe94157b60.slice/crio-b71f2ad24aa440f6ee41cb9125f1fd5ca453a804d635f01c976c1ae81a83cee8 WatchSource:0}: Error finding container b71f2ad24aa440f6ee41cb9125f1fd5ca453a804d635f01c976c1ae81a83cee8: Status 404 returned error can't find the container with id b71f2ad24aa440f6ee41cb9125f1fd5ca453a804d635f01c976c1ae81a83cee8
Sep 10 14:50:28.257328 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:28.257288    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755c66cc4d-lv4ks" event={"ID":"94f5ae91-8c00-49a1-8963-2bbe94157b60","Type":"ContainerStarted","Data":"b71f2ad24aa440f6ee41cb9125f1fd5ca453a804d635f01c976c1ae81a83cee8"}
Sep 10 14:50:31.272013 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:31.271969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755c66cc4d-lv4ks" event={"ID":"94f5ae91-8c00-49a1-8963-2bbe94157b60","Type":"ContainerStarted","Data":"6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944"}
Sep 10 14:50:31.294590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:31.294540    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-755c66cc4d-lv4ks" podStartSLOduration=1.531270729 podStartE2EDuration="4.294525683s" totalImagesPullingTime="2.763254954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:31.292047454 +0000 UTC m=+501.769107407" watchObservedRunningTime="2026-09-10 14:50:31.294525683 +0000 UTC m=+501.771585624"
Sep 10 14:50:31.321464 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:31.321430    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8m7zn"]
Sep 10 14:50:31.321672 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:31.321641    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-8m7zn" podUID="d1ce94f8-7a11-4614-ae5a-63175864be61" containerName="authorino" containerID="cri-o://9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f" gracePeriod=30
Sep 10 14:50:31.592341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:31.592315    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8m7zn"
Sep 10 14:50:31.665055 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:31.665028    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hfqc\" (UniqueName: \"kubernetes.io/projected/d1ce94f8-7a11-4614-ae5a-63175864be61-kube-api-access-4hfqc\") pod \"d1ce94f8-7a11-4614-ae5a-63175864be61\" (UID: \"d1ce94f8-7a11-4614-ae5a-63175864be61\") "
Sep 10 14:50:31.667032 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:31.667002    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ce94f8-7a11-4614-ae5a-63175864be61-kube-api-access-4hfqc" (OuterVolumeSpecName: "kube-api-access-4hfqc") pod "d1ce94f8-7a11-4614-ae5a-63175864be61" (UID: "d1ce94f8-7a11-4614-ae5a-63175864be61"). InnerVolumeSpecName "kube-api-access-4hfqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:50:31.766024 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:31.765991    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hfqc\" (UniqueName: \"kubernetes.io/projected/d1ce94f8-7a11-4614-ae5a-63175864be61-kube-api-access-4hfqc\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:50:32.277848 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:32.277810    2563 generic.go:356] "Generic (PLEG): container finished" podID="d1ce94f8-7a11-4614-ae5a-63175864be61" containerID="9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f" exitCode=0
Sep 10 14:50:32.278244 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:32.277847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8m7zn" event={"ID":"d1ce94f8-7a11-4614-ae5a-63175864be61","Type":"ContainerDied","Data":"9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f"}
Sep 10 14:50:32.278244 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:32.277855    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8m7zn"
Sep 10 14:50:32.278244 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:32.277896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8m7zn" event={"ID":"d1ce94f8-7a11-4614-ae5a-63175864be61","Type":"ContainerDied","Data":"fcd8b132d96e33d296a54e7885402601e22e3018f96086caaf8983ab77c5bb08"}
Sep 10 14:50:32.278244 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:32.277920    2563 scope.go:117] "RemoveContainer" containerID="9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f"
Sep 10 14:50:32.286316 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:32.286289    2563 scope.go:117] "RemoveContainer" containerID="9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f"
Sep 10 14:50:32.286593 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:50:32.286574    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f\": container with ID starting with 9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f not found: ID does not exist" containerID="9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f"
Sep 10 14:50:32.286671 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:32.286601    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f"} err="failed to get container status \"9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f\": rpc error: code = NotFound desc = could not find container \"9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f\": container with ID starting with 9f0a6f2964abd51ae1e9794ec54622eadac8e089662c83b295619480afd44f1f not found: ID does not exist"
Sep 10 14:50:32.293035 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:32.292984    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8m7zn"]
Sep 10 14:50:32.296910 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:32.296889    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8m7zn"]
Sep 10 14:50:33.724146 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:33.724114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:34.031771 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:34.031694    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1ce94f8-7a11-4614-ae5a-63175864be61" path="/var/lib/kubelet/pods/d1ce94f8-7a11-4614-ae5a-63175864be61/volumes"
Sep 10 14:50:36.180886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.180849    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:36.209549 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.209496    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-rjcx7"]
Sep 10 14:50:36.209886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.209872    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1ce94f8-7a11-4614-ae5a-63175864be61" containerName="authorino"
Sep 10 14:50:36.210220 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.209888    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1ce94f8-7a11-4614-ae5a-63175864be61" containerName="authorino"
Sep 10 14:50:36.210220 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.209953    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1ce94f8-7a11-4614-ae5a-63175864be61" containerName="authorino"
Sep 10 14:50:36.212110 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.212090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.214336 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.214312    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 14:50:36.214442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.214340    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhrqr\""
Sep 10 14:50:36.214442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.214357    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 14:50:36.214784 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.214769    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 14:50:36.225275 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.225251    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-rjcx7"]
Sep 10 14:50:36.281475 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.281444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:36.292680 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.292656    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-webhook-cert\") pod \"maas-controller-7bbdd8d44-rjcx7\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") " pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.292794 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.292693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jfkt\" (UniqueName: \"kubernetes.io/projected/7b211eee-62e7-49a1-8998-019d90a11f69-kube-api-access-6jfkt\") pod \"maas-controller-7bbdd8d44-rjcx7\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") " pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.292794 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.292762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-metrics-certs\") pod \"maas-controller-7bbdd8d44-rjcx7\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") " pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.384463 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.384432    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:36.393390 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.393095    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-metrics-certs\") pod \"maas-controller-7bbdd8d44-rjcx7\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") " pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.393390 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.393209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-webhook-cert\") pod \"maas-controller-7bbdd8d44-rjcx7\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") " pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.393390 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.393241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jfkt\" (UniqueName: \"kubernetes.io/projected/7b211eee-62e7-49a1-8998-019d90a11f69-kube-api-access-6jfkt\") pod \"maas-controller-7bbdd8d44-rjcx7\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") " pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.394935 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.394914    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-2ghvs"]
Sep 10 14:50:36.395778 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.395755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-webhook-cert\") pod \"maas-controller-7bbdd8d44-rjcx7\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") " pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.395778 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.395769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-metrics-certs\") pod \"maas-controller-7bbdd8d44-rjcx7\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") " pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.402397 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.402375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.410934 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.410915    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-2ghvs"]
Sep 10 14:50:36.412470 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.412447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jfkt\" (UniqueName: \"kubernetes.io/projected/7b211eee-62e7-49a1-8998-019d90a11f69-kube-api-access-6jfkt\") pod \"maas-controller-7bbdd8d44-rjcx7\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") " pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.453298 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.453232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:36.493848 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.493824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bjz\" (UniqueName: \"kubernetes.io/projected/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-kube-api-access-k8bjz\") pod \"maas-controller-5bcc5b4d4d-2ghvs\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.493965 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.493890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-metrics-certs\") pod \"maas-controller-5bcc5b4d4d-2ghvs\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.493965 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.493921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-webhook-cert\") pod \"maas-controller-5bcc5b4d4d-2ghvs\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.522122 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.522098    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:36.595243 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.595207    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bjz\" (UniqueName: \"kubernetes.io/projected/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-kube-api-access-k8bjz\") pod \"maas-controller-5bcc5b4d4d-2ghvs\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.595373 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.595305    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-metrics-certs\") pod \"maas-controller-5bcc5b4d4d-2ghvs\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.595373 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.595341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-webhook-cert\") pod \"maas-controller-5bcc5b4d4d-2ghvs\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.597687 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.597663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-metrics-certs\") pod \"maas-controller-5bcc5b4d4d-2ghvs\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.598167 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.598149    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-webhook-cert\") pod \"maas-controller-5bcc5b4d4d-2ghvs\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.602545 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.602524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bjz\" (UniqueName: \"kubernetes.io/projected/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-kube-api-access-k8bjz\") pod \"maas-controller-5bcc5b4d4d-2ghvs\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.730839 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.730745    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:36.791129 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:36.791098    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-rjcx7"]
Sep 10 14:50:36.794161 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:50:36.794125    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b211eee_62e7_49a1_8998_019d90a11f69.slice/crio-1403b9cc6a0f7bd365b8f34efd33d465b66d9a5840e2ae1269eba5a3e8edb680 WatchSource:0}: Error finding container 1403b9cc6a0f7bd365b8f34efd33d465b66d9a5840e2ae1269eba5a3e8edb680: Status 404 returned error can't find the container with id 1403b9cc6a0f7bd365b8f34efd33d465b66d9a5840e2ae1269eba5a3e8edb680
Sep 10 14:50:37.001113 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:37.001043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-2ghvs"]
Sep 10 14:50:37.004059 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:50:37.004037    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed2fdf5_3f55_4033_90c0_1fac2531d4e1.slice/crio-6d2579099cfe2cbcf03790d8ff167837ddecca4bf1a9789a3625f6902e87c76d WatchSource:0}: Error finding container 6d2579099cfe2cbcf03790d8ff167837ddecca4bf1a9789a3625f6902e87c76d: Status 404 returned error can't find the container with id 6d2579099cfe2cbcf03790d8ff167837ddecca4bf1a9789a3625f6902e87c76d
Sep 10 14:50:37.302777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:37.302663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs" event={"ID":"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1","Type":"ContainerStarted","Data":"6d2579099cfe2cbcf03790d8ff167837ddecca4bf1a9789a3625f6902e87c76d"}
Sep 10 14:50:37.303875 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:37.303849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7" event={"ID":"7b211eee-62e7-49a1-8998-019d90a11f69","Type":"ContainerStarted","Data":"1403b9cc6a0f7bd365b8f34efd33d465b66d9a5840e2ae1269eba5a3e8edb680"}
Sep 10 14:50:38.587590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:38.587556    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:38.688758 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:38.688668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:50:39.314904 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:39.314873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7" event={"ID":"7b211eee-62e7-49a1-8998-019d90a11f69","Type":"ContainerStarted","Data":"b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4"}
Sep 10 14:50:39.315109 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:39.314932    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:39.316411 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:39.316382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs" event={"ID":"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1","Type":"ContainerStarted","Data":"149beb688e7955d3c331331bdd4ceb6d8ad050e92f7242970e452ab3e45aac3a"}
Sep 10 14:50:39.316564 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:39.316500    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:39.332401 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:39.332361    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7" podStartSLOduration=1.724579017 podStartE2EDuration="3.332346853s" totalImagesPullingTime="1.607767836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:39.330587482 +0000 UTC m=+509.807647426" watchObservedRunningTime="2026-09-10 14:50:39.332346853 +0000 UTC m=+509.809406799"
Sep 10 14:50:39.347466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:39.347428    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs" podStartSLOduration=1.945568658 podStartE2EDuration="3.347414106s" totalImagesPullingTime="1.401845448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:50:39.346888885 +0000 UTC m=+509.823948830" watchObservedRunningTime="2026-09-10 14:50:39.347414106 +0000 UTC m=+509.824474050"
Sep 10 14:50:50.324604 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.324572    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:50:50.325059 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.324656    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:50.385801 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.385766    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-rjcx7"]
Sep 10 14:50:50.386023 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.385997    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7" podUID="7b211eee-62e7-49a1-8998-019d90a11f69" containerName="manager" containerID="cri-o://b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4" gracePeriod=30
Sep 10 14:50:50.622428 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.622406    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:50.718141 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.718109    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jfkt\" (UniqueName: \"kubernetes.io/projected/7b211eee-62e7-49a1-8998-019d90a11f69-kube-api-access-6jfkt\") pod \"7b211eee-62e7-49a1-8998-019d90a11f69\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") "
Sep 10 14:50:50.718141 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.718143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-webhook-cert\") pod \"7b211eee-62e7-49a1-8998-019d90a11f69\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") "
Sep 10 14:50:50.718353 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.718195    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-metrics-certs\") pod \"7b211eee-62e7-49a1-8998-019d90a11f69\" (UID: \"7b211eee-62e7-49a1-8998-019d90a11f69\") "
Sep 10 14:50:50.720367 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.720334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b211eee-62e7-49a1-8998-019d90a11f69" (UID: "7b211eee-62e7-49a1-8998-019d90a11f69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:50.720488 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.720366    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7b211eee-62e7-49a1-8998-019d90a11f69" (UID: "7b211eee-62e7-49a1-8998-019d90a11f69"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:50:50.720488 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.720389    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b211eee-62e7-49a1-8998-019d90a11f69-kube-api-access-6jfkt" (OuterVolumeSpecName: "kube-api-access-6jfkt") pod "7b211eee-62e7-49a1-8998-019d90a11f69" (UID: "7b211eee-62e7-49a1-8998-019d90a11f69"). InnerVolumeSpecName "kube-api-access-6jfkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:50:50.818945 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.818914    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jfkt\" (UniqueName: \"kubernetes.io/projected/7b211eee-62e7-49a1-8998-019d90a11f69-kube-api-access-6jfkt\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:50:50.818945 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.818941    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-webhook-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:50:50.818945 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:50.818950    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b211eee-62e7-49a1-8998-019d90a11f69-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:50:51.377602 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:51.377567    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b211eee-62e7-49a1-8998-019d90a11f69" containerID="b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4" exitCode=0
Sep 10 14:50:51.378052 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:51.377611    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7"
Sep 10 14:50:51.378052 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:51.377654    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7" event={"ID":"7b211eee-62e7-49a1-8998-019d90a11f69","Type":"ContainerDied","Data":"b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4"}
Sep 10 14:50:51.378052 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:51.377715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbdd8d44-rjcx7" event={"ID":"7b211eee-62e7-49a1-8998-019d90a11f69","Type":"ContainerDied","Data":"1403b9cc6a0f7bd365b8f34efd33d465b66d9a5840e2ae1269eba5a3e8edb680"}
Sep 10 14:50:51.378052 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:51.377749    2563 scope.go:117] "RemoveContainer" containerID="b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4"
Sep 10 14:50:51.387442 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:51.387246    2563 scope.go:117] "RemoveContainer" containerID="b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4"
Sep 10 14:50:51.387606 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:50:51.387548    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4\": container with ID starting with b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4 not found: ID does not exist" containerID="b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4"
Sep 10 14:50:51.387606 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:51.387572    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4"} err="failed to get container status \"b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4\": rpc error: code = NotFound desc = could not find container \"b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4\": container with ID starting with b26ff90ff8b39e09797bcc7eec1cf8ffe6a4fbded3e2491f4c47764c97e14df4 not found: ID does not exist"
Sep 10 14:50:51.398463 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:51.398413    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-rjcx7"]
Sep 10 14:50:51.402181 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:51.402164    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bbdd8d44-rjcx7"]
Sep 10 14:50:52.032093 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:50:52.032060    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b211eee-62e7-49a1-8998-019d90a11f69" path="/var/lib/kubelet/pods/7b211eee-62e7-49a1-8998-019d90a11f69/volumes"
Sep 10 14:51:11.219526 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:11.219458    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:11.311031 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:11.310998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:12.703864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.703828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:12.801186 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.801147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:12.832291 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.832262    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"]
Sep 10 14:51:12.832680 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.832642    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b211eee-62e7-49a1-8998-019d90a11f69" containerName="manager"
Sep 10 14:51:12.832732 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.832681    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b211eee-62e7-49a1-8998-019d90a11f69" containerName="manager"
Sep 10 14:51:12.832768 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.832758    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b211eee-62e7-49a1-8998-019d90a11f69" containerName="manager"
Sep 10 14:51:12.838400 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.838380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:12.841364 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.841341    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 14:51:12.841534 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.841375    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9t8qm\""
Sep 10 14:51:12.841534 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.841375    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 14:51:12.845015 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.844993    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"]
Sep 10 14:51:12.901436 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:12.901399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:13.012713 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.012625    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfrc9\" (UniqueName: \"kubernetes.io/projected/053b0de1-9a92-4797-bfa6-d9653a09ee0e-kube-api-access-sfrc9\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.012713 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.012673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-maas-api-tls\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.012713 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.012708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-metrics-certs\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.012941 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.012761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053b0de1-9a92-4797-bfa6-d9653a09ee0e-openshift-service-ca\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.101279 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.101247    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:13.113315 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.113287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfrc9\" (UniqueName: \"kubernetes.io/projected/053b0de1-9a92-4797-bfa6-d9653a09ee0e-kube-api-access-sfrc9\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.113483 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.113332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-maas-api-tls\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.113483 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.113353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-metrics-certs\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.113483 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.113396    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053b0de1-9a92-4797-bfa6-d9653a09ee0e-openshift-service-ca\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.114197 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.114169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053b0de1-9a92-4797-bfa6-d9653a09ee0e-openshift-service-ca\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.115986 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.115962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-maas-api-tls\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.116063 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.115968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-metrics-certs\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.120696 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.120673    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfrc9\" (UniqueName: \"kubernetes.io/projected/053b0de1-9a92-4797-bfa6-d9653a09ee0e-kube-api-access-sfrc9\") pod \"maas-api-7f975847c4-rjvs2\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") " pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.150704 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.150677    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:13.290992 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.290918    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:13.391369 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.391338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:13.432025 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.431996    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"]
Sep 10 14:51:13.435455 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:51:13.435405    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053b0de1_9a92_4797_bfa6_d9653a09ee0e.slice/crio-d9551adda3bd2d552f860b516e1bccf21f6c65e8037b20d98f3a5190d4f315c7 WatchSource:0}: Error finding container d9551adda3bd2d552f860b516e1bccf21f6c65e8037b20d98f3a5190d4f315c7: Status 404 returned error can't find the container with id d9551adda3bd2d552f860b516e1bccf21f6c65e8037b20d98f3a5190d4f315c7
Sep 10 14:51:13.480479 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:13.480447    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2" event={"ID":"053b0de1-9a92-4797-bfa6-d9653a09ee0e","Type":"ContainerStarted","Data":"d9551adda3bd2d552f860b516e1bccf21f6c65e8037b20d98f3a5190d4f315c7"}
Sep 10 14:51:15.492354 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:15.492323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2" event={"ID":"053b0de1-9a92-4797-bfa6-d9653a09ee0e","Type":"ContainerStarted","Data":"8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac"}
Sep 10 14:51:15.492817 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:15.492446    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:15.508874 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:15.508829    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2" podStartSLOduration=2.010259981 podStartE2EDuration="3.508814117s" totalImagesPullingTime="1.498554136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:51:15.508325355 +0000 UTC m=+545.985385300" watchObservedRunningTime="2026-09-10 14:51:15.508814117 +0000 UTC m=+545.985874129"
Sep 10 14:51:21.502578 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:21.502547    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 14:51:22.490676 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:22.490609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:22.594839 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:22.594808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:30.108160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:30.108121    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:42.365264 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.365230    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"]
Sep 10 14:51:42.369865 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.369836    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.372849 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.372823    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 14:51:42.373007 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.372911    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 14:51:42.373007 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.372822    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5tt4z\""
Sep 10 14:51:42.373142 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.372829    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 14:51:42.379367 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.379346    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"]
Sep 10 14:51:42.396721 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.396692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:51:42.454497 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.454461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.454497 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.454495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.454758 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.454606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w5rf\" (UniqueName: \"kubernetes.io/projected/13bd8514-e3ac-410b-8454-cd619c144491-kube-api-access-6w5rf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.454758 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.454663    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.454758 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.454689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bd8514-e3ac-410b-8454-cd619c144491-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.454758 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.454714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.555604 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.555573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.555604 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.555607    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.555819 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.555640    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w5rf\" (UniqueName: \"kubernetes.io/projected/13bd8514-e3ac-410b-8454-cd619c144491-kube-api-access-6w5rf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.555819 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.555680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.555819 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.555696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bd8514-e3ac-410b-8454-cd619c144491-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.555819 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.555715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.556373 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.556052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.556373 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.556087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.556373 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.556275    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.558122 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.558102    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13bd8514-e3ac-410b-8454-cd619c144491-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.558430 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.558413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13bd8514-e3ac-410b-8454-cd619c144491-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.563334 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.563310    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w5rf\" (UniqueName: \"kubernetes.io/projected/13bd8514-e3ac-410b-8454-cd619c144491-kube-api-access-6w5rf\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4zl47\" (UID: \"13bd8514-e3ac-410b-8454-cd619c144491\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.682864 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.682831    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:42.955017 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:42.954939    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"]
Sep 10 14:51:42.958033 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:51:42.958011    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bd8514_e3ac_410b_8454_cd619c144491.slice/crio-30b55fb8c0617b0968d98aeb065a7f6c9c07b7a8bf5682a1780b95382d864c68 WatchSource:0}: Error finding container 30b55fb8c0617b0968d98aeb065a7f6c9c07b7a8bf5682a1780b95382d864c68: Status 404 returned error can't find the container with id 30b55fb8c0617b0968d98aeb065a7f6c9c07b7a8bf5682a1780b95382d864c68
Sep 10 14:51:43.632478 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:43.632434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47" event={"ID":"13bd8514-e3ac-410b-8454-cd619c144491","Type":"ContainerStarted","Data":"30b55fb8c0617b0968d98aeb065a7f6c9c07b7a8bf5682a1780b95382d864c68"}
Sep 10 14:51:44.641014 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:44.640971    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47" event={"ID":"13bd8514-e3ac-410b-8454-cd619c144491","Type":"ContainerStarted","Data":"ac5ed32c68700c89ed134bf1bfbd8443e07c39e1f820657ae0f35d0637a7d7b2"}
Sep 10 14:51:44.659010 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:44.658940    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47" podStartSLOduration=1.609358088 podStartE2EDuration="2.658919342s" totalImagesPullingTime="1.049561254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:51:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:51:44.658330072 +0000 UTC m=+575.135390033" watchObservedRunningTime="2026-09-10 14:51:44.658919342 +0000 UTC m=+575.135979287"
Sep 10 14:51:52.683501 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:52.683467    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:52.683901 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:52.683554    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:52.695793 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:52.695769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:51:53.695109 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:51:53.695082    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4zl47"
Sep 10 14:52:07.106846 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:07.106808    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:52:09.973363 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:09.973337    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 14:52:09.973971 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:09.973954    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 14:52:23.065159 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.065125    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"]
Sep 10 14:52:23.067555 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.067538    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.069533 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.069489    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 10 14:52:23.077926 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.077904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"]
Sep 10 14:52:23.095188 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.095156    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8hfb\" (UniqueName: \"kubernetes.io/projected/754462ae-275d-442d-950f-6539698e2440-kube-api-access-b8hfb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.095303 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.095254    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.095303 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.095285    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.095415 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.095305    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754462ae-275d-442d-950f-6539698e2440-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.095415 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.095332    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.095578 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.095558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.159952 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.159915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:52:23.197027 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.196998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.197160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.197055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8hfb\" (UniqueName: \"kubernetes.io/projected/754462ae-275d-442d-950f-6539698e2440-kube-api-access-b8hfb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.197160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.197090    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.197160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.197109    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.197160 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.197130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754462ae-275d-442d-950f-6539698e2440-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.197364 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.197284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.197447 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.197418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.197531 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.197468    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.197619 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.197603    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.199445 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.199415    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754462ae-275d-442d-950f-6539698e2440-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.199781 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.199765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754462ae-275d-442d-950f-6539698e2440-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.204886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.204865    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8hfb\" (UniqueName: \"kubernetes.io/projected/754462ae-275d-442d-950f-6539698e2440-kube-api-access-b8hfb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8vhkm\" (UID: \"754462ae-275d-442d-950f-6539698e2440\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.378394 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.378316    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:23.654582 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.654558    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"]
Sep 10 14:52:23.657307 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:52:23.657278    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754462ae_275d_442d_950f_6539698e2440.slice/crio-c922e536848ab5b96ef52ef46aea3307f6a8eea8a0224a8475e32cf579491859 WatchSource:0}: Error finding container c922e536848ab5b96ef52ef46aea3307f6a8eea8a0224a8475e32cf579491859: Status 404 returned error can't find the container with id c922e536848ab5b96ef52ef46aea3307f6a8eea8a0224a8475e32cf579491859
Sep 10 14:52:23.829484 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:23.829446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm" event={"ID":"754462ae-275d-442d-950f-6539698e2440","Type":"ContainerStarted","Data":"c922e536848ab5b96ef52ef46aea3307f6a8eea8a0224a8475e32cf579491859"}
Sep 10 14:52:24.837577 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:24.837536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm" event={"ID":"754462ae-275d-442d-950f-6539698e2440","Type":"ContainerStarted","Data":"f5667d1f79057720ad1425c90ff9e7055696954abb3b33992d8033ead22b3398"}
Sep 10 14:52:24.857136 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:24.857074    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm" podStartSLOduration=1.648081371 podStartE2EDuration="1.857051427s" totalImagesPullingTime="208.970056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:52:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:52:24.854020205 +0000 UTC m=+615.331080142" watchObservedRunningTime="2026-09-10 14:52:24.857051427 +0000 UTC m=+615.334111372"
Sep 10 14:52:25.577122 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:25.577077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:52:33.379313 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:33.379224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:33.379313 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:33.379302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:33.391601 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:33.391577    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:33.891786 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:33.891756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8vhkm"
Sep 10 14:52:38.098275 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:52:38.098239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:53:02.002925 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:53:02.002887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:53:08.492269 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:53:08.492238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:53:40.115194 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:53:40.115158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:54:17.536968 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:17.536886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:54:37.610888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:37.610846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:54:42.951625 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:42.951581    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-755c66cc4d-lv4ks"]
Sep 10 14:54:42.952197 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:42.951815    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-755c66cc4d-lv4ks" podUID="94f5ae91-8c00-49a1-8963-2bbe94157b60" containerName="authorino" containerID="cri-o://6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944" gracePeriod=30
Sep 10 14:54:43.205776 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.205722    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:54:43.320202 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.320172    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzr8z\" (UniqueName: \"kubernetes.io/projected/94f5ae91-8c00-49a1-8963-2bbe94157b60-kube-api-access-gzr8z\") pod \"94f5ae91-8c00-49a1-8963-2bbe94157b60\" (UID: \"94f5ae91-8c00-49a1-8963-2bbe94157b60\") "
Sep 10 14:54:43.320418 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.320224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94f5ae91-8c00-49a1-8963-2bbe94157b60-tls-cert\") pod \"94f5ae91-8c00-49a1-8963-2bbe94157b60\" (UID: \"94f5ae91-8c00-49a1-8963-2bbe94157b60\") "
Sep 10 14:54:43.322434 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.322401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f5ae91-8c00-49a1-8963-2bbe94157b60-kube-api-access-gzr8z" (OuterVolumeSpecName: "kube-api-access-gzr8z") pod "94f5ae91-8c00-49a1-8963-2bbe94157b60" (UID: "94f5ae91-8c00-49a1-8963-2bbe94157b60"). InnerVolumeSpecName "kube-api-access-gzr8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:54:43.330466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.330443    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f5ae91-8c00-49a1-8963-2bbe94157b60-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "94f5ae91-8c00-49a1-8963-2bbe94157b60" (UID: "94f5ae91-8c00-49a1-8963-2bbe94157b60"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:54:43.421131 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.421100    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/94f5ae91-8c00-49a1-8963-2bbe94157b60-tls-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:54:43.421131 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.421125    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzr8z\" (UniqueName: \"kubernetes.io/projected/94f5ae91-8c00-49a1-8963-2bbe94157b60-kube-api-access-gzr8z\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:54:43.462888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.462752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:54:43.495794 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.495758    2563 generic.go:356] "Generic (PLEG): container finished" podID="94f5ae91-8c00-49a1-8963-2bbe94157b60" containerID="6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944" exitCode=0
Sep 10 14:54:43.495937 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.495808    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-755c66cc4d-lv4ks"
Sep 10 14:54:43.495937 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.495844    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755c66cc4d-lv4ks" event={"ID":"94f5ae91-8c00-49a1-8963-2bbe94157b60","Type":"ContainerDied","Data":"6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944"}
Sep 10 14:54:43.495937 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.495889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-755c66cc4d-lv4ks" event={"ID":"94f5ae91-8c00-49a1-8963-2bbe94157b60","Type":"ContainerDied","Data":"b71f2ad24aa440f6ee41cb9125f1fd5ca453a804d635f01c976c1ae81a83cee8"}
Sep 10 14:54:43.495937 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.495907    2563 scope.go:117] "RemoveContainer" containerID="6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944"
Sep 10 14:54:43.506635 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.506596    2563 scope.go:117] "RemoveContainer" containerID="6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944"
Sep 10 14:54:43.507012 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:54:43.506991    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944\": container with ID starting with 6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944 not found: ID does not exist" containerID="6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944"
Sep 10 14:54:43.507080 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.507020    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944"} err="failed to get container status \"6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944\": rpc error: code = NotFound desc = could not find container \"6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944\": container with ID starting with 6ca3cee5c0446528dc2e9f2249e60497b66ce694b25334a1aa176bfda79e4944 not found: ID does not exist"
Sep 10 14:54:43.534484 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.534441    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-755c66cc4d-lv4ks"]
Sep 10 14:54:43.535814 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.535790    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-755c66cc4d-lv4ks"]
Sep 10 14:54:43.546607 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:43.546581    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:54:44.031884 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:44.031853    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f5ae91-8c00-49a1-8963-2bbe94157b60" path="/var/lib/kubelet/pods/94f5ae91-8c00-49a1-8963-2bbe94157b60/volumes"
Sep 10 14:54:44.611910 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:44.611876    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:54:44.706150 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:54:44.706124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:01.045022 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:01.044969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:01.165906 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:01.165872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:01.408951 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:01.408914    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:01.560840 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:01.560810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:02.551325 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:02.551290    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:02.655883 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:02.655850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:03.043975 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.043921    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:03.165667 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.165637    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7668-dd707d"]
Sep 10 14:55:03.166034 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.166022    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f5ae91-8c00-49a1-8963-2bbe94157b60" containerName="authorino"
Sep 10 14:55:03.166079 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.166037    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f5ae91-8c00-49a1-8963-2bbe94157b60" containerName="authorino"
Sep 10 14:55:03.166127 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.166118    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f5ae91-8c00-49a1-8963-2bbe94157b60" containerName="authorino"
Sep 10 14:55:03.170488 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.170472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7668-dd707d"
Sep 10 14:55:03.173727 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.173708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mmqf2\""
Sep 10 14:55:03.182786 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.182762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7668-dd707d"]
Sep 10 14:55:03.194327 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.194303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkpz7\" (UniqueName: \"kubernetes.io/projected/233862a2-9f41-4d3a-8229-08d05bf46c33-kube-api-access-gkpz7\") pod \"test-curl-7668-dd707d\" (UID: \"233862a2-9f41-4d3a-8229-08d05bf46c33\") " pod="opendatahub/test-curl-7668-dd707d"
Sep 10 14:55:03.295816 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.295732    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkpz7\" (UniqueName: \"kubernetes.io/projected/233862a2-9f41-4d3a-8229-08d05bf46c33-kube-api-access-gkpz7\") pod \"test-curl-7668-dd707d\" (UID: \"233862a2-9f41-4d3a-8229-08d05bf46c33\") " pod="opendatahub/test-curl-7668-dd707d"
Sep 10 14:55:03.310092 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.310062    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkpz7\" (UniqueName: \"kubernetes.io/projected/233862a2-9f41-4d3a-8229-08d05bf46c33-kube-api-access-gkpz7\") pod \"test-curl-7668-dd707d\" (UID: \"233862a2-9f41-4d3a-8229-08d05bf46c33\") " pod="opendatahub/test-curl-7668-dd707d"
Sep 10 14:55:03.479825 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.479799    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7668-dd707d"
Sep 10 14:55:03.749845 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.749822    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7668-dd707d"]
Sep 10 14:55:03.752389 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:55:03.752364    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod233862a2_9f41_4d3a_8229_08d05bf46c33.slice/crio-075a47a069cfc435c7a4939616b85acedd75bd1b1675d8a362a572f9c923db87 WatchSource:0}: Error finding container 075a47a069cfc435c7a4939616b85acedd75bd1b1675d8a362a572f9c923db87: Status 404 returned error can't find the container with id 075a47a069cfc435c7a4939616b85acedd75bd1b1675d8a362a572f9c923db87
Sep 10 14:55:03.754222 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:03.754207    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 14:55:04.358108 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:04.358070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:04.459230 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:04.459202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:04.604041 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:04.604010    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7668-dd707d" event={"ID":"233862a2-9f41-4d3a-8229-08d05bf46c33","Type":"ContainerStarted","Data":"075a47a069cfc435c7a4939616b85acedd75bd1b1675d8a362a572f9c923db87"}
Sep 10 14:55:05.524121 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.523581    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"]
Sep 10 14:55:05.529253 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.529229    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.531987 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.531961    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-74eee9-serving-cert\""
Sep 10 14:55:05.538986 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.538934    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"]
Sep 10 14:55:05.560900 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.560831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:05.615602 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.615573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0905ed37-e70e-4ebd-acf4-3602153bc2eb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.615825 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.615645    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-metrics-certs\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.615825 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.615705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.615825 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.615752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dn9t\" (UniqueName: \"kubernetes.io/projected/0905ed37-e70e-4ebd-acf4-3602153bc2eb-kube-api-access-6dn9t\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.716457 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.716416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dn9t\" (UniqueName: \"kubernetes.io/projected/0905ed37-e70e-4ebd-acf4-3602153bc2eb-kube-api-access-6dn9t\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.716696 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.716544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0905ed37-e70e-4ebd-acf4-3602153bc2eb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.716696 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.716571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-metrics-certs\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.716696 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.716620    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.716867 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:05.716755    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-serving-cert: secret "maas-api-e2e-inf-a-74eee9-serving-cert" not found
Sep 10 14:55:05.716867 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:05.716817    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-maas-api-tls podName:0905ed37-e70e-4ebd-acf4-3602153bc2eb nodeName:}" failed. No retries permitted until 2026-09-10 14:55:06.216799134 +0000 UTC m=+776.693859056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-maas-api-tls") pod "maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5" (UID: "0905ed37-e70e-4ebd-acf4-3602153bc2eb") : secret "maas-api-e2e-inf-a-74eee9-serving-cert" not found
Sep 10 14:55:05.717805 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.717769    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0905ed37-e70e-4ebd-acf4-3602153bc2eb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.719978 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.719953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-metrics-certs\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.729924 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.729898    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dn9t\" (UniqueName: \"kubernetes.io/projected/0905ed37-e70e-4ebd-acf4-3602153bc2eb-kube-api-access-6dn9t\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:05.797452 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:05.797372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:06.220737 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:06.220698    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:06.223192 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:06.223160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-maas-api-tls\") pod \"maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:06.443836 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:06.443794    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:06.617482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:06.617446    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7668-dd707d" event={"ID":"233862a2-9f41-4d3a-8229-08d05bf46c33","Type":"ContainerStarted","Data":"8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac"}
Sep 10 14:55:06.712644 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:06.712579    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7668-dd707d" podStartSLOduration=1.240690035 podStartE2EDuration="3.712558602s" totalImagesPullingTime="2.471868567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:06.630998231 +0000 UTC m=+777.108058181" watchObservedRunningTime="2026-09-10 14:55:06.712558602 +0000 UTC m=+777.189618546"
Sep 10 14:55:06.714304 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:06.714177    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"]
Sep 10 14:55:06.716461 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:55:06.716439    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0905ed37_e70e_4ebd_acf4_3602153bc2eb.slice/crio-035de8470216d20ad40b34e15c211f218158b5e99566c394819af26a9334143f WatchSource:0}: Error finding container 035de8470216d20ad40b34e15c211f218158b5e99566c394819af26a9334143f: Status 404 returned error can't find the container with id 035de8470216d20ad40b34e15c211f218158b5e99566c394819af26a9334143f
Sep 10 14:55:07.242980 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.242947    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7668-dd707d"]
Sep 10 14:55:07.247187 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.247157    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7668-dd707d"]
Sep 10 14:55:07.390043 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.390016    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7668-c80444"]
Sep 10 14:55:07.390427 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.390414    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="233862a2-9f41-4d3a-8229-08d05bf46c33" containerName="test-curl-7668-dd707d"
Sep 10 14:55:07.390467 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.390429    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="233862a2-9f41-4d3a-8229-08d05bf46c33" containerName="test-curl-7668-dd707d"
Sep 10 14:55:07.390502 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.390493    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="233862a2-9f41-4d3a-8229-08d05bf46c33" containerName="test-curl-7668-dd707d"
Sep 10 14:55:07.393394 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.393380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7668-c80444"
Sep 10 14:55:07.403375 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.403355    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7668-c80444"]
Sep 10 14:55:07.431272 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.431233    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slv54\" (UniqueName: \"kubernetes.io/projected/12e7aa32-9506-4f62-af20-15d37e8de254-kube-api-access-slv54\") pod \"test-curl-7668-c80444\" (UID: \"12e7aa32-9506-4f62-af20-15d37e8de254\") " pod="opendatahub/test-curl-7668-c80444"
Sep 10 14:55:07.532334 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.532296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slv54\" (UniqueName: \"kubernetes.io/projected/12e7aa32-9506-4f62-af20-15d37e8de254-kube-api-access-slv54\") pod \"test-curl-7668-c80444\" (UID: \"12e7aa32-9506-4f62-af20-15d37e8de254\") " pod="opendatahub/test-curl-7668-c80444"
Sep 10 14:55:07.541828 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.541798    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slv54\" (UniqueName: \"kubernetes.io/projected/12e7aa32-9506-4f62-af20-15d37e8de254-kube-api-access-slv54\") pod \"test-curl-7668-c80444\" (UID: \"12e7aa32-9506-4f62-af20-15d37e8de254\") " pod="opendatahub/test-curl-7668-c80444"
Sep 10 14:55:07.623525 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.623472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5" event={"ID":"0905ed37-e70e-4ebd-acf4-3602153bc2eb","Type":"ContainerStarted","Data":"9acbda0b049967776f62323ac6659bda7171fd939a7c4becec39c60354b3daf5"}
Sep 10 14:55:07.623896 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.623536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5" event={"ID":"0905ed37-e70e-4ebd-acf4-3602153bc2eb","Type":"ContainerStarted","Data":"035de8470216d20ad40b34e15c211f218158b5e99566c394819af26a9334143f"}
Sep 10 14:55:07.623896 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.623698    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:07.623896 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:07.623701    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7668-dd707d" container="test-curl-7668-dd707d"
Sep 10 14:55:07.643127 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.643019    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5" podStartSLOduration=2.332368019 podStartE2EDuration="2.643002811s" totalImagesPullingTime="310.634792ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:07.641211239 +0000 UTC m=+778.118271184" watchObservedRunningTime="2026-09-10 14:55:07.643002811 +0000 UTC m=+778.120062755"
Sep 10 14:55:07.702240 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.702211    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7668-c80444"
Sep 10 14:55:07.975550 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:07.975501    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7668-c80444"]
Sep 10 14:55:07.977952 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:55:07.977928    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12e7aa32_9506_4f62_af20_15d37e8de254.slice/crio-9dcc5fbd31ed9b0300778c39eb00c6a9509b808f5e8bf69fd9873f15da9321c2 WatchSource:0}: Error finding container 9dcc5fbd31ed9b0300778c39eb00c6a9509b808f5e8bf69fd9873f15da9321c2: Status 404 returned error can't find the container with id 9dcc5fbd31ed9b0300778c39eb00c6a9509b808f5e8bf69fd9873f15da9321c2
Sep 10 14:55:08.630670 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:08.630639    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7668-c80444" event={"ID":"12e7aa32-9506-4f62-af20-15d37e8de254","Type":"ContainerStarted","Data":"7ba43eaa59dfe2f2a8aa925ba2082b407f4e3b3086ec9d0449bd3caeaa332e47"}
Sep 10 14:55:08.631100 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:08.630680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7668-c80444" event={"ID":"12e7aa32-9506-4f62-af20-15d37e8de254","Type":"ContainerStarted","Data":"9dcc5fbd31ed9b0300778c39eb00c6a9509b808f5e8bf69fd9873f15da9321c2"}
Sep 10 14:55:08.631100 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:08.630922    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7668-dd707d" podUID="233862a2-9f41-4d3a-8229-08d05bf46c33" containerName="test-curl-7668-dd707d" containerID="cri-o://8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac" gracePeriod=2
Sep 10 14:55:08.647553 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:08.647486    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7668-c80444" podStartSLOduration=1.6474694250000002 podStartE2EDuration="1.647469425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:08.647441058 +0000 UTC m=+779.124501002" watchObservedRunningTime="2026-09-10 14:55:08.647469425 +0000 UTC m=+779.124529389"
Sep 10 14:55:09.212722 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:09.212671    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:09.244212 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:09.244180    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7668-c80444"]
Sep 10 14:55:09.249733 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:09.249706    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7668-c80444"]
Sep 10 14:55:09.313161 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:09.313061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:09.443414 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:09.443382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:09.635139 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:09.635061    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7668-c80444" container="test-curl-7668-c80444"
Sep 10 14:55:09.650207 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:09.650179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:09.758382 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:09.758347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:09.822353 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:09.822313    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:10.162643 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.162613    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"]
Sep 10 14:55:10.162998 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.162986    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e7aa32-9506-4f62-af20-15d37e8de254" containerName="test-curl-7668-c80444"
Sep 10 14:55:10.163049 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.163000    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e7aa32-9506-4f62-af20-15d37e8de254" containerName="test-curl-7668-c80444"
Sep 10 14:55:10.163084 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.163051    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e7aa32-9506-4f62-af20-15d37e8de254" containerName="test-curl-7668-c80444"
Sep 10 14:55:10.167448 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.167432    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.170546 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.170522    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 14:55:10.170750 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.170737    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-4454b7-openshift-default-dockercfg-kp6xw\""
Sep 10 14:55:10.183712 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.183691    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"]
Sep 10 14:55:10.200950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.200887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:10.261720 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.261691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-data\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.261720 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.261724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-credential-socket\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.261915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.261792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-certs\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.261915 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.261864    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vdjm\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-kube-api-access-8vdjm\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.261983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.261922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-socket\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.261983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.261945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-token\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.261983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.261968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istiod-ca-cert\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.262087 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.262032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-envoy\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.262087 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.262060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-podinfo\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.300346 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.300269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:10.362663 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.362634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vdjm\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-kube-api-access-8vdjm\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.362858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.362670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-socket\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.362858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.362694    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-token\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.362858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.362712    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istiod-ca-cert\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.362858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.362742    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-envoy\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.362858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.362759    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-podinfo\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.362858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.362789    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-data\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.362858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.362815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-credential-socket\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.362858 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.362858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-certs\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.363271 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.363126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-socket\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.363271 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.363199    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-data\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.363271 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.363229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-certs\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.363410 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.363389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-credential-socket\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.363707 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.363689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istiod-ca-cert\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.365041 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.365016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-envoy\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.365202 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.365182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-podinfo\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.371472 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.371446    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-token\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.371624 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.371604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vdjm\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-kube-api-access-8vdjm\") pod \"e2e-autores-4454b7-openshift-default-5fb7686554-pldch\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") " pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.400369 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.400340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:10.478189 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.478161    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:10.641182 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.641128    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7668-c80444" podUID="12e7aa32-9506-4f62-af20-15d37e8de254" containerName="test-curl-7668-c80444" containerID="cri-o://7ba43eaa59dfe2f2a8aa925ba2082b407f4e3b3086ec9d0449bd3caeaa332e47" gracePeriod=2
Sep 10 14:55:10.754430 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.754407    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"]
Sep 10 14:55:10.757029 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:55:10.757002    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1dbb5e1_b331_4cc3_8e2b_4e49569f201f.slice/crio-a43a7b5a8dcd8a6b8ec4dd8fb8d7c24a871847a172c029d97b59bf0baf27454a WatchSource:0}: Error finding container a43a7b5a8dcd8a6b8ec4dd8fb8d7c24a871847a172c029d97b59bf0baf27454a: Status 404 returned error can't find the container with id a43a7b5a8dcd8a6b8ec4dd8fb8d7c24a871847a172c029d97b59bf0baf27454a
Sep 10 14:55:10.801838 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.801811    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7668-dd707d"
Sep 10 14:55:10.867894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.867866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkpz7\" (UniqueName: \"kubernetes.io/projected/233862a2-9f41-4d3a-8229-08d05bf46c33-kube-api-access-gkpz7\") pod \"233862a2-9f41-4d3a-8229-08d05bf46c33\" (UID: \"233862a2-9f41-4d3a-8229-08d05bf46c33\") "
Sep 10 14:55:10.870044 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.870021    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233862a2-9f41-4d3a-8229-08d05bf46c33-kube-api-access-gkpz7" (OuterVolumeSpecName: "kube-api-access-gkpz7") pod "233862a2-9f41-4d3a-8229-08d05bf46c33" (UID: "233862a2-9f41-4d3a-8229-08d05bf46c33"). InnerVolumeSpecName "kube-api-access-gkpz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:10.969173 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:10.969137    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkpz7\" (UniqueName: \"kubernetes.io/projected/233862a2-9f41-4d3a-8229-08d05bf46c33-kube-api-access-gkpz7\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:11.647286 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:11.647251    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch" event={"ID":"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f","Type":"ContainerStarted","Data":"a43a7b5a8dcd8a6b8ec4dd8fb8d7c24a871847a172c029d97b59bf0baf27454a"}
Sep 10 14:55:11.648550 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:11.648497    2563 generic.go:356] "Generic (PLEG): container finished" podID="233862a2-9f41-4d3a-8229-08d05bf46c33" containerID="8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac" exitCode=137
Sep 10 14:55:11.648687 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:11.648556    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7668-dd707d"
Sep 10 14:55:11.648687 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:11.648570    2563 scope.go:117] "RemoveContainer" containerID="8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac"
Sep 10 14:55:11.656935 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:11.656920    2563 scope.go:117] "RemoveContainer" containerID="8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac"
Sep 10 14:55:11.657212 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:11.657184    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac\": container with ID starting with 8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac not found: ID does not exist" containerID="8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac"
Sep 10 14:55:11.657267 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:11.657220    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac"} err="failed to get container status \"8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac\": rpc error: code = NotFound desc = could not find container \"8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac\": container with ID starting with 8f3f450d36daddca034ca7a348f884987235717001ebf12fd5f08a807ff1f1ac not found: ID does not exist"
Sep 10 14:55:11.657920 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:11.657895    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7668-dd707d\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="233862a2-9f41-4d3a-8229-08d05bf46c33" pod="opendatahub/test-curl-7668-dd707d"
Sep 10 14:55:12.034454 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:12.034376    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="233862a2-9f41-4d3a-8229-08d05bf46c33" path="/var/lib/kubelet/pods/233862a2-9f41-4d3a-8229-08d05bf46c33/volumes"
Sep 10 14:55:13.637810 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:13.637777    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:55:13.665777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:13.665658    2563 generic.go:356] "Generic (PLEG): container finished" podID="12e7aa32-9506-4f62-af20-15d37e8de254" containerID="7ba43eaa59dfe2f2a8aa925ba2082b407f4e3b3086ec9d0449bd3caeaa332e47" exitCode=137
Sep 10 14:55:13.761732 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:13.761709    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7668-c80444"
Sep 10 14:55:13.805211 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:13.805182    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:55:13.805293 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:13.805247    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:55:13.805293 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:13.805275    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:55:13.899679 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:13.899654    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slv54\" (UniqueName: \"kubernetes.io/projected/12e7aa32-9506-4f62-af20-15d37e8de254-kube-api-access-slv54\") pod \"12e7aa32-9506-4f62-af20-15d37e8de254\" (UID: \"12e7aa32-9506-4f62-af20-15d37e8de254\") "
Sep 10 14:55:13.901678 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:13.901648    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e7aa32-9506-4f62-af20-15d37e8de254-kube-api-access-slv54" (OuterVolumeSpecName: "kube-api-access-slv54") pod "12e7aa32-9506-4f62-af20-15d37e8de254" (UID: "12e7aa32-9506-4f62-af20-15d37e8de254"). InnerVolumeSpecName "kube-api-access-slv54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:14.000929 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:14.000889    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slv54\" (UniqueName: \"kubernetes.io/projected/12e7aa32-9506-4f62-af20-15d37e8de254-kube-api-access-slv54\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:14.031406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:14.031379    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e7aa32-9506-4f62-af20-15d37e8de254" path="/var/lib/kubelet/pods/12e7aa32-9506-4f62-af20-15d37e8de254/volumes"
Sep 10 14:55:14.671318 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:14.671295    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7668-c80444"
Sep 10 14:55:14.671810 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:14.671334    2563 scope.go:117] "RemoveContainer" containerID="7ba43eaa59dfe2f2a8aa925ba2082b407f4e3b3086ec9d0449bd3caeaa332e47"
Sep 10 14:55:14.673302 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:14.673268    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch" event={"ID":"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f","Type":"ContainerStarted","Data":"4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5"}
Sep 10 14:55:14.693670 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:14.693623    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch" podStartSLOduration=1.6480256660000001 podStartE2EDuration="4.693608245s" totalImagesPullingTime="3.045582579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:14.692683856 +0000 UTC m=+785.169743791" watchObservedRunningTime="2026-09-10 14:55:14.693608245 +0000 UTC m=+785.170668189"
Sep 10 14:55:15.478339 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:15.478307    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:15.483664 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:15.483636    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:15.678641 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:15.678612    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:15.679754 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:15.679738    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:17.333729 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:17.333645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:17.580937 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:17.580904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:17.689819 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:17.689783    2563 generic.go:356] "Generic (PLEG): container finished" podID="37acfe05-5214-4191-abb7-741d7d0e46b4" containerID="b032c1182616831b02f22e0fb5c39081457592fc619af472f5d0c3bd1cef7208" exitCode=1
Sep 10 14:55:17.689975 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:17.689853    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j" event={"ID":"37acfe05-5214-4191-abb7-741d7d0e46b4","Type":"ContainerDied","Data":"b032c1182616831b02f22e0fb5c39081457592fc619af472f5d0c3bd1cef7208"}
Sep 10 14:55:17.690288 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:17.690271    2563 scope.go:117] "RemoveContainer" containerID="b032c1182616831b02f22e0fb5c39081457592fc619af472f5d0c3bd1cef7208"
Sep 10 14:55:17.691644 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:17.691622    2563 generic.go:356] "Generic (PLEG): container finished" podID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" containerID="149beb688e7955d3c331331bdd4ceb6d8ad050e92f7242970e452ab3e45aac3a" exitCode=1
Sep 10 14:55:17.691732 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:17.691700    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs" event={"ID":"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1","Type":"ContainerDied","Data":"149beb688e7955d3c331331bdd4ceb6d8ad050e92f7242970e452ab3e45aac3a"}
Sep 10 14:55:17.692071 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:17.692053    2563 scope.go:117] "RemoveContainer" containerID="149beb688e7955d3c331331bdd4ceb6d8ad050e92f7242970e452ab3e45aac3a"
Sep 10 14:55:17.923840 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:17.923811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:18.025092 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.025015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:18.048758 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.045458    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"]
Sep 10 14:55:18.130902 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.130871    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:18.223876 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.223836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:18.326754 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.326649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:18.450156 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.450124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:18.618500 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.618394    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:18.698567 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.698533    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j" event={"ID":"37acfe05-5214-4191-abb7-741d7d0e46b4","Type":"ContainerStarted","Data":"4762a11ea29c20665a86298a9c05fe5e9dce16e664bd63e3212cef8c12594ea0"}
Sep 10 14:55:18.698773 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.698756    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:55:18.700383 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.700360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs" event={"ID":"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1","Type":"ContainerStarted","Data":"3c0ff3ad6761ca3a7f3d8dabc50f5b0bc2840fc7ae2586c7a3e02e19acbb52d7"}
Sep 10 14:55:18.700580 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.700432    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch" podUID="f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" containerName="istio-proxy" containerID="cri-o://4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5" gracePeriod=30
Sep 10 14:55:18.700580 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.700568    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:55:18.729249 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.729219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:18.909294 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.909257    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"]
Sep 10 14:55:18.913484 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.913460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:18.916383 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.916360    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-bae22d-openshift-default-dockercfg-2k9gn\""
Sep 10 14:55:18.931090 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.931068    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"]
Sep 10 14:55:18.941226 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:18.941203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:19.046704 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.046674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-certs\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.046886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.046725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-data\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.046886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.046743    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmwf\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-kube-api-access-dfmwf\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.046886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.046827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-socket\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.046886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.046854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-token\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.046886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.046877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-envoy\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.047068 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.046900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f89f10-ecc7-4aa5-8f58-042cb098e913-istiod-ca-cert\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.047068 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.046964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-credential-socket\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.047068 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.046998    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-podinfo\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.085140 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.085108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:19.147887 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.147858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-certs\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148065 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.147907    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-data\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148065 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.147925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmwf\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-kube-api-access-dfmwf\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148180 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-socket\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148180 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-token\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148180 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-envoy\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148180 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f89f10-ecc7-4aa5-8f58-042cb098e913-istiod-ca-cert\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148402 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-credential-socket\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148402 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-podinfo\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148402 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148316    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-certs\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148597 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148434    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-socket\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148656 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-credential-socket\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148852 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-data\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.148988 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.148936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f89f10-ecc7-4aa5-8f58-042cb098e913-istiod-ca-cert\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.150420 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.150402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-envoy\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.150623 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.150608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-podinfo\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.155597 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.155567    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-token\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.156157 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.156135    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmwf\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-kube-api-access-dfmwf\") pod \"e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") " pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.225429 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.225344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:19.503964 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.503887    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"]
Sep 10 14:55:19.507555 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:55:19.507528    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f89f10_ecc7_4aa5_8f58_042cb098e913.slice/crio-642fc26d79408fb7690b618553390d985077d621dc9b6b148e379db65fa9c746 WatchSource:0}: Error finding container 642fc26d79408fb7690b618553390d985077d621dc9b6b148e379db65fa9c746: Status 404 returned error can't find the container with id 642fc26d79408fb7690b618553390d985077d621dc9b6b148e379db65fa9c746
Sep 10 14:55:19.511194 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.511154    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:55:19.511325 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.511224    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:55:19.511325 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.511250    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:55:19.707581 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.707543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n" event={"ID":"67f89f10-ecc7-4aa5-8f58-042cb098e913","Type":"ContainerStarted","Data":"6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821"}
Sep 10 14:55:19.707787 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.707612    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n" event={"ID":"67f89f10-ecc7-4aa5-8f58-042cb098e913","Type":"ContainerStarted","Data":"642fc26d79408fb7690b618553390d985077d621dc9b6b148e379db65fa9c746"}
Sep 10 14:55:19.728172 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:19.728108    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n" podStartSLOduration=1.72808921 podStartE2EDuration="1.72808921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:19.72632396 +0000 UTC m=+790.203383904" watchObservedRunningTime="2026-09-10 14:55:19.72808921 +0000 UTC m=+790.205149154"
Sep 10 14:55:20.021899 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:20.021865    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:20.226429 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:20.226401    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:20.228131 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:20.228093    2563 patch_prober.go:28] interesting pod/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Sep 10 14:55:20.228286 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:20.228169    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n" podUID="67f89f10-ecc7-4aa5-8f58-042cb098e913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Sep 10 14:55:20.237078 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:20.236899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:20.467029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:20.466999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:20.812749 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:20.812627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:20.909167 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:20.909087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:21.230679 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:21.230645    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:21.497439 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:21.497359    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"]
Sep 10 14:55:21.525841 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:21.525812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:21.719363 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:21.719327    2563 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n" secret="" err="secret \"e2e-explicit-bae22d-openshift-default-dockercfg-2k9gn\" not found"
Sep 10 14:55:21.719563 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:21.719343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:21.720208 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:21.720189    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:21.745865 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:21.745838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:21.855713 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:21.855641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:22.068919 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:22.068875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:22.724077 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:22.724034    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n" podUID="67f89f10-ecc7-4aa5-8f58-042cb098e913" containerName="istio-proxy" containerID="cri-o://6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821" gracePeriod=30
Sep 10 14:55:23.960923 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:23.960856    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:24.093777 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.093746    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-token\") pod \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") "
Sep 10 14:55:24.093962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.093828    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-data\") pod \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") "
Sep 10 14:55:24.093962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.093877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-socket\") pod \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") "
Sep 10 14:55:24.093962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.093903    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vdjm\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-kube-api-access-8vdjm\") pod \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") "
Sep 10 14:55:24.093962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.093930    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-envoy\") pod \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") "
Sep 10 14:55:24.093962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.093947    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-certs\") pod \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") "
Sep 10 14:55:24.094232 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094004    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istiod-ca-cert\") pod \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") "
Sep 10 14:55:24.094232 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094030    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-podinfo\") pod \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") "
Sep 10 14:55:24.094232 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-credential-socket\") pod \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\" (UID: \"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f\") "
Sep 10 14:55:24.094232 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" (UID: "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:24.094232 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094149    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" (UID: "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:24.094482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094343    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" (UID: "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:24.094482 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094468    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-credential-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:24.094643 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094486    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-data\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:24.094643 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094500    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:24.094643 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094465    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" (UID: "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:24.094643 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.094487    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" (UID: "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:55:24.096089 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.096058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" (UID: "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:24.096209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.096082    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-kube-api-access-8vdjm" (OuterVolumeSpecName: "kube-api-access-8vdjm") pod "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" (UID: "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f"). InnerVolumeSpecName "kube-api-access-8vdjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:24.096209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.096152    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" (UID: "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:55:24.096295 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.096239    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" (UID: "f1dbb5e1-b331-4cc3-8e2b-4e49569f201f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:24.196004 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.195971    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vdjm\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-kube-api-access-8vdjm\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:24.196004 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.196006    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-envoy\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:24.196209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.196021    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-workload-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:24.196209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.196032    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istiod-ca-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:24.196209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.196044    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-podinfo\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:24.196209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.196055    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f-istio-token\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:24.695710 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.695672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:24.735410 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.735379    2563 generic.go:356] "Generic (PLEG): container finished" podID="f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" containerID="4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5" exitCode=0
Sep 10 14:55:24.735590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.735429    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"
Sep 10 14:55:24.735590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.735443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch" event={"ID":"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f","Type":"ContainerDied","Data":"4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5"}
Sep 10 14:55:24.735590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.735484    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch" event={"ID":"f1dbb5e1-b331-4cc3-8e2b-4e49569f201f","Type":"ContainerDied","Data":"a43a7b5a8dcd8a6b8ec4dd8fb8d7c24a871847a172c029d97b59bf0baf27454a"}
Sep 10 14:55:24.735590 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.735521    2563 scope.go:117] "RemoveContainer" containerID="4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5"
Sep 10 14:55:24.745109 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.745089    2563 scope.go:117] "RemoveContainer" containerID="4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5"
Sep 10 14:55:24.745359 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:24.745343    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5\": container with ID starting with 4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5 not found: ID does not exist" containerID="4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5"
Sep 10 14:55:24.745408 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.745367    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5"} err="failed to get container status \"4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5\": rpc error: code = NotFound desc = could not find container \"4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5\": container with ID starting with 4bb2a8c8e054d5151d0157d030e782a7711231b4efba3905f14ac40559bb86a5 not found: ID does not exist"
Sep 10 14:55:24.757711 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.757689    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"]
Sep 10 14:55:24.762649 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.762628    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-4454b7-openshift-default-5fb7686554-pldch"]
Sep 10 14:55:24.860464 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:24.860417    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:25.065438 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:25.065356    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:25.182067 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:25.182036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:25.274039 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:25.274004    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:25.372309 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:25.372225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:25.571362 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:25.571326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:26.032192 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:26.032154    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" path="/var/lib/kubelet/pods/f1dbb5e1-b331-4cc3-8e2b-4e49569f201f/volumes"
Sep 10 14:55:26.096276 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:26.096245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:27.983706 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:27.983646    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:28.131898 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.131866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-credential-socket\") pod \"67f89f10-ecc7-4aa5-8f58-042cb098e913\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") "
Sep 10 14:55:28.132093 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.131906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-socket\") pod \"67f89f10-ecc7-4aa5-8f58-042cb098e913\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") "
Sep 10 14:55:28.132093 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.131939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f89f10-ecc7-4aa5-8f58-042cb098e913-istiod-ca-cert\") pod \"67f89f10-ecc7-4aa5-8f58-042cb098e913\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") "
Sep 10 14:55:28.132093 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.131966    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-podinfo\") pod \"67f89f10-ecc7-4aa5-8f58-042cb098e913\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") "
Sep 10 14:55:28.132093 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132005    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-certs\") pod \"67f89f10-ecc7-4aa5-8f58-042cb098e913\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") "
Sep 10 14:55:28.132299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132171    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-token\") pod \"67f89f10-ecc7-4aa5-8f58-042cb098e913\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") "
Sep 10 14:55:28.132299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132235    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-data\") pod \"67f89f10-ecc7-4aa5-8f58-042cb098e913\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") "
Sep 10 14:55:28.132299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67f89f10-ecc7-4aa5-8f58-042cb098e913" (UID: "67f89f10-ecc7-4aa5-8f58-042cb098e913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:28.132299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-envoy\") pod \"67f89f10-ecc7-4aa5-8f58-042cb098e913\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") "
Sep 10 14:55:28.132299 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132190    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67f89f10-ecc7-4aa5-8f58-042cb098e913" (UID: "67f89f10-ecc7-4aa5-8f58-042cb098e913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:28.132600 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132319    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67f89f10-ecc7-4aa5-8f58-042cb098e913" (UID: "67f89f10-ecc7-4aa5-8f58-042cb098e913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:28.132600 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfmwf\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-kube-api-access-dfmwf\") pod \"67f89f10-ecc7-4aa5-8f58-042cb098e913\" (UID: \"67f89f10-ecc7-4aa5-8f58-042cb098e913\") "
Sep 10 14:55:28.132600 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132418    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f89f10-ecc7-4aa5-8f58-042cb098e913-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67f89f10-ecc7-4aa5-8f58-042cb098e913" (UID: "67f89f10-ecc7-4aa5-8f58-042cb098e913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:55:28.132600 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132574    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-data" (OuterVolumeSpecName: "istio-data") pod "67f89f10-ecc7-4aa5-8f58-042cb098e913" (UID: "67f89f10-ecc7-4aa5-8f58-042cb098e913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:28.132798 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132753    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:28.132798 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132772    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-data\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:28.132798 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132785    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-credential-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:28.132922 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132799    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-workload-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:28.132922 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.132813    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67f89f10-ecc7-4aa5-8f58-042cb098e913-istiod-ca-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:28.134234 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.134202    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67f89f10-ecc7-4aa5-8f58-042cb098e913" (UID: "67f89f10-ecc7-4aa5-8f58-042cb098e913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:55:28.134367 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.134314    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67f89f10-ecc7-4aa5-8f58-042cb098e913" (UID: "67f89f10-ecc7-4aa5-8f58-042cb098e913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:55:28.134705 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.134680    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-token" (OuterVolumeSpecName: "istio-token") pod "67f89f10-ecc7-4aa5-8f58-042cb098e913" (UID: "67f89f10-ecc7-4aa5-8f58-042cb098e913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:28.134705 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.134678    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-kube-api-access-dfmwf" (OuterVolumeSpecName: "kube-api-access-dfmwf") pod "67f89f10-ecc7-4aa5-8f58-042cb098e913" (UID: "67f89f10-ecc7-4aa5-8f58-042cb098e913"). InnerVolumeSpecName "kube-api-access-dfmwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:55:28.233453 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.233422    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-podinfo\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:28.233453 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.233447    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-token\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:28.233453 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.233457    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67f89f10-ecc7-4aa5-8f58-042cb098e913-istio-envoy\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:28.233681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.233465    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfmwf\" (UniqueName: \"kubernetes.io/projected/67f89f10-ecc7-4aa5-8f58-042cb098e913-kube-api-access-dfmwf\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:55:28.694256 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.694217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:28.756996 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.756964    2563 generic.go:356] "Generic (PLEG): container finished" podID="67f89f10-ecc7-4aa5-8f58-042cb098e913" containerID="6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821" exitCode=0
Sep 10 14:55:28.757177 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.757012    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"
Sep 10 14:55:28.757177 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.757020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n" event={"ID":"67f89f10-ecc7-4aa5-8f58-042cb098e913","Type":"ContainerDied","Data":"6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821"}
Sep 10 14:55:28.757177 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.757070    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n" event={"ID":"67f89f10-ecc7-4aa5-8f58-042cb098e913","Type":"ContainerDied","Data":"642fc26d79408fb7690b618553390d985077d621dc9b6b148e379db65fa9c746"}
Sep 10 14:55:28.757177 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.757094    2563 scope.go:117] "RemoveContainer" containerID="6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821"
Sep 10 14:55:28.766362 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.766346    2563 scope.go:117] "RemoveContainer" containerID="6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821"
Sep 10 14:55:28.766683 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:28.766664    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821\": container with ID starting with 6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821 not found: ID does not exist" containerID="6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821"
Sep 10 14:55:28.766766 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.766688    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821"} err="failed to get container status \"6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821\": rpc error: code = NotFound desc = could not find container \"6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821\": container with ID starting with 6cb5888f4077748d3a0d4bc35d2278405cc28dcdfb9f63dc0b0c730453e47821 not found: ID does not exist"
Sep 10 14:55:28.777988 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.777968    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"]
Sep 10 14:55:28.781878 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:28.781860    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-bae22d-openshift-default-84785dc564-r5w9n"]
Sep 10 14:55:29.710352 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:29.710320    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 14:55:30.032851 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:30.032772    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f89f10-ecc7-4aa5-8f58-042cb098e913" path="/var/lib/kubelet/pods/67f89f10-ecc7-4aa5-8f58-042cb098e913/volumes"
Sep 10 14:55:31.252980 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:31.252948    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:31.331183 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:31.331154    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:32.426621 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.426545    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"]
Sep 10 14:55:32.426980 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.426969    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" containerName="istio-proxy"
Sep 10 14:55:32.427026 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.426982    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" containerName="istio-proxy"
Sep 10 14:55:32.427026 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.426998    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f89f10-ecc7-4aa5-8f58-042cb098e913" containerName="istio-proxy"
Sep 10 14:55:32.427026 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.427003    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f89f10-ecc7-4aa5-8f58-042cb098e913" containerName="istio-proxy"
Sep 10 14:55:32.427134 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.427067    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1dbb5e1-b331-4cc3-8e2b-4e49569f201f" containerName="istio-proxy"
Sep 10 14:55:32.427134 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.427077    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f89f10-ecc7-4aa5-8f58-042cb098e913" containerName="istio-proxy"
Sep 10 14:55:32.433717 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.433697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.437581 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.437559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 14:55:32.437682 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.437625    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-bd601b-openshift-default-dockercfg-bvblt\""
Sep 10 14:55:32.453876 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.453853    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"]
Sep 10 14:55:32.490350 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.490315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:32.568701 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.568665    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-envoy\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.568701 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.568701    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-istio-token\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.568952 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.568734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-data\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.568952 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.568792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-socket\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.568952 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.568866    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3704233d-8315-4622-b315-7d4f32337b5d-istio-podinfo\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.568952 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.568900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-certs\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.569127 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.568964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db7sb\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-kube-api-access-db7sb\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.569127 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.569032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3704233d-8315-4622-b315-7d4f32337b5d-istiod-ca-cert\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.569127 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.569056    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-credential-socket\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.667347 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.667314    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:32.670405 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-data\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670481 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670417    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-socket\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670481 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3704233d-8315-4622-b315-7d4f32337b5d-istio-podinfo\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670481 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-certs\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670636 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db7sb\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-kube-api-access-db7sb\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670636 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3704233d-8315-4622-b315-7d4f32337b5d-istiod-ca-cert\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670636 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-credential-socket\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670788 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-envoy\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670788 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-istio-token\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670900 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-data\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.670900 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-certs\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.671012 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.670988    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-socket\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.671071 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.671014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-credential-socket\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.671368 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.671346    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3704233d-8315-4622-b315-7d4f32337b5d-istiod-ca-cert\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.672839 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.672823    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-envoy\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.672926 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.672860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3704233d-8315-4622-b315-7d4f32337b5d-istio-podinfo\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.679903 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.679847    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-istio-token\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.681358 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.681327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db7sb\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-kube-api-access-db7sb\") pod \"e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") " pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:32.744134 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:32.744108    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:33.023930 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:33.023904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"]
Sep 10 14:55:33.026288 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:55:33.026260    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3704233d_8315_4622_b315_7d4f32337b5d.slice/crio-549005bcd4a8903f9e55335dc16239daf7840d421c4cba654af2fb61f4a2cbe8 WatchSource:0}: Error finding container 549005bcd4a8903f9e55335dc16239daf7840d421c4cba654af2fb61f4a2cbe8: Status 404 returned error can't find the container with id 549005bcd4a8903f9e55335dc16239daf7840d421c4cba654af2fb61f4a2cbe8
Sep 10 14:55:33.029456 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:33.029423    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:55:33.029562 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:33.029493    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:55:33.029562 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:33.029547    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:55:33.784851 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:33.784815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv" event={"ID":"3704233d-8315-4622-b315-7d4f32337b5d","Type":"ContainerStarted","Data":"2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73"}
Sep 10 14:55:33.785233 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:33.784866    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv" event={"ID":"3704233d-8315-4622-b315-7d4f32337b5d","Type":"ContainerStarted","Data":"549005bcd4a8903f9e55335dc16239daf7840d421c4cba654af2fb61f4a2cbe8"}
Sep 10 14:55:33.806403 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:33.806357    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv" podStartSLOduration=1.80634183 podStartE2EDuration="1.80634183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:33.804202962 +0000 UTC m=+804.281262896" watchObservedRunningTime="2026-09-10 14:55:33.80634183 +0000 UTC m=+804.283401773"
Sep 10 14:55:34.102501 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:34.102421    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:34.196866 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:34.196829    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:34.298638 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:34.298606    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:34.364345 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:34.364258    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:34.745059 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:34.745026    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:34.750781 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:34.750757    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:34.789555 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:34.789528    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:34.790459 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:34.790441    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:55:36.927359 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:36.927326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:37.153306 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:37.153275    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:37.714364 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:37.714326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:37.808804 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:37.808738    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:39.380718 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:39.380685    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:39.560925 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:39.560888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:39.921491 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:39.921459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:40.022493 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.022457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:40.050736 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.050709    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"]
Sep 10 14:55:40.054226 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.054202    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:40.057108 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.057086    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-74eee9\""
Sep 10 14:55:40.082022 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.081998    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"]
Sep 10 14:55:40.139752 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.139721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxp7\" (UniqueName: \"kubernetes.io/projected/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-kube-api-access-jqxp7\") pod \"payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz\" (UID: \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:40.139930 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.139787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz\" (UID: \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:40.240682 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.240612    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxp7\" (UniqueName: \"kubernetes.io/projected/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-kube-api-access-jqxp7\") pod \"payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz\" (UID: \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:40.240682 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.240668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz\" (UID: \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:40.241275 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.241259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz\" (UID: \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:40.262906 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.262881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxp7\" (UniqueName: \"kubernetes.io/projected/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-kube-api-access-jqxp7\") pod \"payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz\" (UID: \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\") " pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:40.366012 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.365968    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:40.641549 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.641502    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"]
Sep 10 14:55:40.643615 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:55:40.643585    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5aaa67_abde_4b8b_b901_39de7daf0ea0.slice/crio-8ac032acff8c78af0a923afceffe144363037d4ac7d14a9fbe58ee0459597c58 WatchSource:0}: Error finding container 8ac032acff8c78af0a923afceffe144363037d4ac7d14a9fbe58ee0459597c58: Status 404 returned error can't find the container with id 8ac032acff8c78af0a923afceffe144363037d4ac7d14a9fbe58ee0459597c58
Sep 10 14:55:40.821251 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:40.821217    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz" event={"ID":"fd5aaa67-abde-4b8b-b901-39de7daf0ea0","Type":"ContainerStarted","Data":"8ac032acff8c78af0a923afceffe144363037d4ac7d14a9fbe58ee0459597c58"}
Sep 10 14:55:43.840541 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:43.840492    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz" event={"ID":"fd5aaa67-abde-4b8b-b901-39de7daf0ea0","Type":"ContainerStarted","Data":"72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759"}
Sep 10 14:55:43.840924 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:43.840657    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:43.859514 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:43.859462    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz" podStartSLOduration=1.529740609 podStartE2EDuration="3.859445227s" totalImagesPullingTime="2.329704618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:43.858038651 +0000 UTC m=+814.335098594" watchObservedRunningTime="2026-09-10 14:55:43.859445227 +0000 UTC m=+814.336505175"
Sep 10 14:55:44.604253 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:44.604199    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:44.913041 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:44.912992    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:45.395914 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:45.395886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:45.805309 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:45.805221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:45.909656 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:45.909621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:46.009157 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:46.009122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:46.108249 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:46.108146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:48.587669 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:48.587621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:48.677418 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:48.677382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:49.715082 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.715052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-q692j"
Sep 10 14:55:49.736534 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.735837    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"]
Sep 10 14:55:49.742708 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.742678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.746189 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.746161    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ff661a-serving-cert\""
Sep 10 14:55:49.750559 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.750533    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"]
Sep 10 14:55:49.782304 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.782269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:49.822604 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.822568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65db817-a615-4746-8441-1f345f37ee5c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.822758 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.822660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-maas-api-tls\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.822758 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.822691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-metrics-certs\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.822835 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.822759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24ql\" (UniqueName: \"kubernetes.io/projected/d65db817-a615-4746-8441-1f345f37ee5c-kube-api-access-v24ql\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.882452 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.882415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:49.923486 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.923445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-maas-api-tls\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.923661 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.923518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-metrics-certs\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.923661 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.923559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24ql\" (UniqueName: \"kubernetes.io/projected/d65db817-a615-4746-8441-1f345f37ee5c-kube-api-access-v24ql\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.923661 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:49.923604    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-serving-cert: secret "maas-api-e2e-inf-b-ff661a-serving-cert" not found
Sep 10 14:55:49.923661 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.923646    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65db817-a615-4746-8441-1f345f37ee5c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.923865 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:55:49.923672    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-maas-api-tls podName:d65db817-a615-4746-8441-1f345f37ee5c nodeName:}" failed. No retries permitted until 2026-09-10 14:55:50.423652191 +0000 UTC m=+820.900712113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-maas-api-tls") pod "maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b" (UID: "d65db817-a615-4746-8441-1f345f37ee5c") : secret "maas-api-e2e-inf-b-ff661a-serving-cert" not found
Sep 10 14:55:49.924414 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.924391    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65db817-a615-4746-8441-1f345f37ee5c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.925986 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.925955    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-metrics-certs\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:49.932605 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:49.932578    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24ql\" (UniqueName: \"kubernetes.io/projected/d65db817-a615-4746-8441-1f345f37ee5c-kube-api-access-v24ql\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:50.189198 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:50.189159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:50.405660 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:50.405627    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:50.428667 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:50.428632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-maas-api-tls\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:50.430962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:50.430932    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-maas-api-tls\") pod \"maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:50.508117 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:50.508030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:50.659635 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:50.659606    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:50.938393 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:50.938367    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"]
Sep 10 14:55:50.940418 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:55:50.940392    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65db817_a615_4746_8441_1f345f37ee5c.slice/crio-99f211a288cc73e59ccd7d23ef124266f441dcfd939133d3888f91209b495977 WatchSource:0}: Error finding container 99f211a288cc73e59ccd7d23ef124266f441dcfd939133d3888f91209b495977: Status 404 returned error can't find the container with id 99f211a288cc73e59ccd7d23ef124266f441dcfd939133d3888f91209b495977
Sep 10 14:55:51.882196 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:51.882152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b" event={"ID":"d65db817-a615-4746-8441-1f345f37ee5c","Type":"ContainerStarted","Data":"386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c"}
Sep 10 14:55:51.882376 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:51.882214    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b" event={"ID":"d65db817-a615-4746-8441-1f345f37ee5c","Type":"ContainerStarted","Data":"99f211a288cc73e59ccd7d23ef124266f441dcfd939133d3888f91209b495977"}
Sep 10 14:55:51.882376 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:51.882312    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:55:51.898349 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:51.898303    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b" podStartSLOduration=2.541598674 podStartE2EDuration="2.898287888s" totalImagesPullingTime="356.689214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:55:51.896457418 +0000 UTC m=+822.373517361" watchObservedRunningTime="2026-09-10 14:55:51.898287888 +0000 UTC m=+822.375347832"
Sep 10 14:55:53.842486 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:53.842453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:53.995663 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:53.995622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:54.118647 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:54.118557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:54.220654 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:54.220614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:54.849458 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:54.849429    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:55:56.551070 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:56.551033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:55:57.892039 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:55:57.892009    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:56:06.936910 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:06.936872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:13.477439 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:13.477404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:13.577816 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:13.577786    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:19.027849 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:19.027801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:22.485389 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:22.485345    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:28.393458 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:28.393410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:28.497421 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:28.497381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:28.638015 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:28.637966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:39.079961 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:39.079926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:44.592538 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:44.592474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:50.421751 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:50.421713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:50.566045 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:50.566002    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:50.664687 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:50.664653    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:56:51.209015 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:56:51.208973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:01.108996 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:01.108954    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:04.864532 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:04.864439    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"]
Sep 10 14:57:04.868544 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:04.868525    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:04.871058 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:04.871035    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-74eee9\"/\"default-dockercfg-r8smv\""
Sep 10 14:57:04.871180 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:04.871036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-74eee9\"/\"test-model-74eee9-kserve-self-signed-certs\""
Sep 10 14:57:04.871180 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:04.871070    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-74eee9\"/\"kube-root-ca.crt\""
Sep 10 14:57:04.871180 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:04.871070    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-74eee9\"/\"openshift-service-ca.crt\""
Sep 10 14:57:04.879327 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:04.879302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"]
Sep 10 14:57:04.891886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:04.891847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:05.009291 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.009258    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9t92\" (UniqueName: \"kubernetes.io/projected/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-kube-api-access-k9t92\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.009473 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.009297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tmp-dir\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.009473 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.009373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tls-certs\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.009473 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.009437    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-home\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.009682 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.009490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-dshm\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.009682 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.009548    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-model-cache\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.110193 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.110158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-model-cache\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.110386 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.110230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9t92\" (UniqueName: \"kubernetes.io/projected/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-kube-api-access-k9t92\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.110386 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.110265    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tmp-dir\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.110386 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.110301    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tls-certs\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.110386 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.110374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-home\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.110734 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.110420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-dshm\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.110734 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.110672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-model-cache\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.110975 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.110948    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tmp-dir\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.111100 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.110980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-home\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.112825 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.112803    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-dshm\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.113340 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.113318    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tls-certs\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.117992 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.117938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9t92\" (UniqueName: \"kubernetes.io/projected/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-kube-api-access-k9t92\") pod \"test-model-74eee9-kserve-55f469dcbc-pvpws\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") " pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.193963 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.193935    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:05.469723 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:05.469629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"]
Sep 10 14:57:05.472351 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:57:05.472325    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8c093a2_5eed_44f3_9a72_6a0e9867a7a1.slice/crio-eb0f767253f9006c0dd5e536a9bc299d5578c82e83434b990fccbf376018895d WatchSource:0}: Error finding container eb0f767253f9006c0dd5e536a9bc299d5578c82e83434b990fccbf376018895d: Status 404 returned error can't find the container with id eb0f767253f9006c0dd5e536a9bc299d5578c82e83434b990fccbf376018895d
Sep 10 14:57:06.274763 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:06.274729    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws" event={"ID":"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1","Type":"ContainerStarted","Data":"92bd66e7581ccd87e079311a9b7728c1aa24c01cd37a1ceb74d042e7b15f131b"}
Sep 10 14:57:06.274763 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:06.274771    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws" event={"ID":"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1","Type":"ContainerStarted","Data":"eb0f767253f9006c0dd5e536a9bc299d5578c82e83434b990fccbf376018895d"}
Sep 10 14:57:06.291566 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:06.291522    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws" podStartSLOduration=2.291494006 podStartE2EDuration="2.291494006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:06.289363542 +0000 UTC m=+896.766423487" watchObservedRunningTime="2026-09-10 14:57:06.291494006 +0000 UTC m=+896.768554024"
Sep 10 14:57:07.210314 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:07.210282    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:07.291588 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:07.291556    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"]
Sep 10 14:57:07.292048 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:07.291800    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv" podUID="3704233d-8315-4622-b315-7d4f32337b5d" containerName="istio-proxy" containerID="cri-o://2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73" gracePeriod=30
Sep 10 14:57:07.318719 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:07.318683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:07.538209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:07.538122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:07.643931 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:07.643898    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:07.928228 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:07.928192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:10.004007 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:10.003973    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 14:57:10.005552 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:10.005531    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 14:57:12.098183 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.098147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:12.548496 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.548475    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:57:12.576188 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576160    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3704233d-8315-4622-b315-7d4f32337b5d-istiod-ca-cert\") pod \"3704233d-8315-4622-b315-7d4f32337b5d\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") "
Sep 10 14:57:12.576332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576198    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-istio-token\") pod \"3704233d-8315-4622-b315-7d4f32337b5d\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") "
Sep 10 14:57:12.576332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-credential-socket\") pod \"3704233d-8315-4622-b315-7d4f32337b5d\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") "
Sep 10 14:57:12.576332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576255    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-certs\") pod \"3704233d-8315-4622-b315-7d4f32337b5d\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") "
Sep 10 14:57:12.576332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576276    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3704233d-8315-4622-b315-7d4f32337b5d-istio-podinfo\") pod \"3704233d-8315-4622-b315-7d4f32337b5d\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") "
Sep 10 14:57:12.576332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-envoy\") pod \"3704233d-8315-4622-b315-7d4f32337b5d\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") "
Sep 10 14:57:12.576650 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-data\") pod \"3704233d-8315-4622-b315-7d4f32337b5d\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") "
Sep 10 14:57:12.576650 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-socket\") pod \"3704233d-8315-4622-b315-7d4f32337b5d\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") "
Sep 10 14:57:12.576650 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576427    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db7sb\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-kube-api-access-db7sb\") pod \"3704233d-8315-4622-b315-7d4f32337b5d\" (UID: \"3704233d-8315-4622-b315-7d4f32337b5d\") "
Sep 10 14:57:12.576650 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576593    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3704233d-8315-4622-b315-7d4f32337b5d" (UID: "3704233d-8315-4622-b315-7d4f32337b5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:57:12.576650 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576593    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3704233d-8315-4622-b315-7d4f32337b5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3704233d-8315-4622-b315-7d4f32337b5d" (UID: "3704233d-8315-4622-b315-7d4f32337b5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:57:12.576922 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576749    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3704233d-8315-4622-b315-7d4f32337b5d-istiod-ca-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:57:12.576922 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576767    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-credential-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:57:12.576922 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576786    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3704233d-8315-4622-b315-7d4f32337b5d" (UID: "3704233d-8315-4622-b315-7d4f32337b5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:57:12.576922 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.576820    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3704233d-8315-4622-b315-7d4f32337b5d" (UID: "3704233d-8315-4622-b315-7d4f32337b5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:57:12.577161 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.577026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "3704233d-8315-4622-b315-7d4f32337b5d" (UID: "3704233d-8315-4622-b315-7d4f32337b5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:57:12.578627 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.578604    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "3704233d-8315-4622-b315-7d4f32337b5d" (UID: "3704233d-8315-4622-b315-7d4f32337b5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:57:12.578775 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.578759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3704233d-8315-4622-b315-7d4f32337b5d" (UID: "3704233d-8315-4622-b315-7d4f32337b5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:57:12.579177 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.579154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3704233d-8315-4622-b315-7d4f32337b5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3704233d-8315-4622-b315-7d4f32337b5d" (UID: "3704233d-8315-4622-b315-7d4f32337b5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 14:57:12.579283 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.579174    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-kube-api-access-db7sb" (OuterVolumeSpecName: "kube-api-access-db7sb") pod "3704233d-8315-4622-b315-7d4f32337b5d" (UID: "3704233d-8315-4622-b315-7d4f32337b5d"). InnerVolumeSpecName "kube-api-access-db7sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:57:12.677752 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.677684    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-istio-token\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:57:12.677752 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.677708    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:57:12.677752 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.677716    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3704233d-8315-4622-b315-7d4f32337b5d-istio-podinfo\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:57:12.677752 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.677724    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-envoy\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:57:12.677752 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.677733    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-istio-data\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:57:12.677752 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.677742    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3704233d-8315-4622-b315-7d4f32337b5d-workload-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:57:12.677752 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:12.677751    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db7sb\" (UniqueName: \"kubernetes.io/projected/3704233d-8315-4622-b315-7d4f32337b5d-kube-api-access-db7sb\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:57:13.228325 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.228294    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:13.313676 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.313642    2563 generic.go:356] "Generic (PLEG): container finished" podID="3704233d-8315-4622-b315-7d4f32337b5d" containerID="2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73" exitCode=0
Sep 10 14:57:13.313850 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.313695    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"
Sep 10 14:57:13.313850 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.313694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv" event={"ID":"3704233d-8315-4622-b315-7d4f32337b5d","Type":"ContainerDied","Data":"2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73"}
Sep 10 14:57:13.313850 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.313759    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv" event={"ID":"3704233d-8315-4622-b315-7d4f32337b5d","Type":"ContainerDied","Data":"549005bcd4a8903f9e55335dc16239daf7840d421c4cba654af2fb61f4a2cbe8"}
Sep 10 14:57:13.313850 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.313790    2563 scope.go:117] "RemoveContainer" containerID="2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73"
Sep 10 14:57:13.323960 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.323944    2563 scope.go:117] "RemoveContainer" containerID="2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73"
Sep 10 14:57:13.324232 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:57:13.324205    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73\": container with ID starting with 2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73 not found: ID does not exist" containerID="2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73"
Sep 10 14:57:13.324297 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.324244    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73"} err="failed to get container status \"2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73\": rpc error: code = NotFound desc = could not find container \"2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73\": container with ID starting with 2e441918af61837997cddbb82ecdf6c65f8c29e85cddc8a3fdeba1318e68ee73 not found: ID does not exist"
Sep 10 14:57:13.336050 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.336025    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"]
Sep 10 14:57:13.339926 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:13.339905    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-bd601b-openshift-default-67cbf44b94-xkmcv"]
Sep 10 14:57:14.032561 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:14.032529    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3704233d-8315-4622-b315-7d4f32337b5d" path="/var/lib/kubelet/pods/3704233d-8315-4622-b315-7d4f32337b5d/volumes"
Sep 10 14:57:14.674984 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:14.674943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:15.194453 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:15.194423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:15.194654 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:15.194464    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:15.213158 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:15.213128    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:15.336957 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:15.336928    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:57:18.154330 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:18.154287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:18.260894 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:18.260850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:18.359903 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:18.359864    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:18.486813 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:18.486725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:19.146830 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:19.146796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:19.240311 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:19.240278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:20.896102 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:20.896072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:26.810572 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:26.810476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:26.876928 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:26.876896    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:27.458816 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:27.458783    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:33.854003 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:33.853966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:33.962550 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:33.962519    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:34.124313 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:34.124227    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:34.392888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:34.392860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:34.454447 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:34.454418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:34.871948 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:34.871877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:35.098842 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:35.098807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:37.871759 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:37.871722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:37.970051 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:37.970022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:39.103498 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.103464    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"]
Sep 10 14:57:39.103893 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.103864    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3704233d-8315-4622-b315-7d4f32337b5d" containerName="istio-proxy"
Sep 10 14:57:39.103893 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.103875    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3704233d-8315-4622-b315-7d4f32337b5d" containerName="istio-proxy"
Sep 10 14:57:39.103972 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.103947    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3704233d-8315-4622-b315-7d4f32337b5d" containerName="istio-proxy"
Sep 10 14:57:39.107297 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.107281    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.109608 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.109587    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-22282287-serving-cert\""
Sep 10 14:57:39.131959 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.131924    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"]
Sep 10 14:57:39.151978 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.151939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:39.220276 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.220244    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-maas-api-tls\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.220445 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.220299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-metrics-certs\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.220445 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.220383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb04e6f-2780-4333-aff1-7e4eefa969ed-openshift-service-ca\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.220445 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.220410    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k7hr\" (UniqueName: \"kubernetes.io/projected/4bb04e6f-2780-4333-aff1-7e4eefa969ed-kube-api-access-5k7hr\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.256928 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.256889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:39.321454 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.321419    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb04e6f-2780-4333-aff1-7e4eefa969ed-openshift-service-ca\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.321657 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.321463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k7hr\" (UniqueName: \"kubernetes.io/projected/4bb04e6f-2780-4333-aff1-7e4eefa969ed-kube-api-access-5k7hr\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.321657 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.321533    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-maas-api-tls\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.321657 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.321584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-metrics-certs\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.321833 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:57:39.321692    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-22282287-serving-cert: secret "maas-api-e2e-mt-22282287-serving-cert" not found
Sep 10 14:57:39.321833 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:57:39.321771    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-maas-api-tls podName:4bb04e6f-2780-4333-aff1-7e4eefa969ed nodeName:}" failed. No retries permitted until 2026-09-10 14:57:39.821749422 +0000 UTC m=+930.298809343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-maas-api-tls") pod "maas-api-e2e-mt-22282287-7d687d759c-dh8zx" (UID: "4bb04e6f-2780-4333-aff1-7e4eefa969ed") : secret "maas-api-e2e-mt-22282287-serving-cert" not found
Sep 10 14:57:39.322089 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.322071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb04e6f-2780-4333-aff1-7e4eefa969ed-openshift-service-ca\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.324000 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.323978    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-metrics-certs\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.330121 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.330097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k7hr\" (UniqueName: \"kubernetes.io/projected/4bb04e6f-2780-4333-aff1-7e4eefa969ed-kube-api-access-5k7hr\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.460423 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.460386    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:39.744927 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.744823    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:39.827370 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.827332    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-maas-api-tls\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.829963 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.829937    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-maas-api-tls\") pod \"maas-api-e2e-mt-22282287-7d687d759c-dh8zx\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:39.843829 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.843803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:39.881294 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.881259    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"]
Sep 10 14:57:39.884997 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.884977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:39.887061 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.887044    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-22282287\""
Sep 10 14:57:39.887167 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.887145    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-22282287-dockercfg-wsbns\""
Sep 10 14:57:39.895621 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.895600    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"]
Sep 10 14:57:39.922168 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.922142    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"]
Sep 10 14:57:39.925726 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.925707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:57:39.936116 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:39.936096    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"]
Sep 10 14:57:40.019501 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.019412    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:40.033520 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.029896    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf52m\" (UniqueName: \"kubernetes.io/projected/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-kube-api-access-tf52m\") pod \"payload-processing-e2e-mt-22282287-6df7d8c677-btjmd\" (UID: \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\") " pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:40.033520 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.029969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84d8z\" (UniqueName: \"kubernetes.io/projected/c7166542-fbbe-4b3e-9889-3563cad23502-kube-api-access-84d8z\") pod \"payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv\" (UID: \"c7166542-fbbe-4b3e-9889-3563cad23502\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:57:40.033520 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.030021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-plugins-config-volume\") pod \"payload-processing-e2e-mt-22282287-6df7d8c677-btjmd\" (UID: \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\") " pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:40.033520 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.030089    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7166542-fbbe-4b3e-9889-3563cad23502-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv\" (UID: \"c7166542-fbbe-4b3e-9889-3563cad23502\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:57:40.130934 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.130901    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7166542-fbbe-4b3e-9889-3563cad23502-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv\" (UID: \"c7166542-fbbe-4b3e-9889-3563cad23502\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:57:40.131372 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.131004    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf52m\" (UniqueName: \"kubernetes.io/projected/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-kube-api-access-tf52m\") pod \"payload-processing-e2e-mt-22282287-6df7d8c677-btjmd\" (UID: \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\") " pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:40.131372 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.131055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8z\" (UniqueName: \"kubernetes.io/projected/c7166542-fbbe-4b3e-9889-3563cad23502-kube-api-access-84d8z\") pod \"payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv\" (UID: \"c7166542-fbbe-4b3e-9889-3563cad23502\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:57:40.131372 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.131098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-plugins-config-volume\") pod \"payload-processing-e2e-mt-22282287-6df7d8c677-btjmd\" (UID: \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\") " pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:40.132054 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.131962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-plugins-config-volume\") pod \"payload-processing-e2e-mt-22282287-6df7d8c677-btjmd\" (UID: \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\") " pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:40.132484 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.132458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7166542-fbbe-4b3e-9889-3563cad23502-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv\" (UID: \"c7166542-fbbe-4b3e-9889-3563cad23502\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:57:40.139773 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.139745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84d8z\" (UniqueName: \"kubernetes.io/projected/c7166542-fbbe-4b3e-9889-3563cad23502-kube-api-access-84d8z\") pod \"payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv\" (UID: \"c7166542-fbbe-4b3e-9889-3563cad23502\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:57:40.140041 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.140025    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf52m\" (UniqueName: \"kubernetes.io/projected/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-kube-api-access-tf52m\") pod \"payload-processing-e2e-mt-22282287-6df7d8c677-btjmd\" (UID: \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\") " pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:40.196162 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.196136    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:40.236869 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.236790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:57:40.318682 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.318488    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"]
Sep 10 14:57:40.323255 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:57:40.323230    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb04e6f_2780_4333_aff1_7e4eefa969ed.slice/crio-368bbd6a249d2f91371e190e97a23092842937cf7f93a753fc05fd0967f206dc WatchSource:0}: Error finding container 368bbd6a249d2f91371e190e97a23092842937cf7f93a753fc05fd0967f206dc: Status 404 returned error can't find the container with id 368bbd6a249d2f91371e190e97a23092842937cf7f93a753fc05fd0967f206dc
Sep 10 14:57:40.456420 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.456389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx" event={"ID":"4bb04e6f-2780-4333-aff1-7e4eefa969ed","Type":"ContainerStarted","Data":"368bbd6a249d2f91371e190e97a23092842937cf7f93a753fc05fd0967f206dc"}
Sep 10 14:57:40.481994 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.481970    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"]
Sep 10 14:57:40.484146 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:57:40.484119    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb5c9fd_dbdb_4574_9d0a_f0546bde3817.slice/crio-a9f3193eef8632efdaead8741525f576abc311fa2aff08f3d4438ffbe26ad92c WatchSource:0}: Error finding container a9f3193eef8632efdaead8741525f576abc311fa2aff08f3d4438ffbe26ad92c: Status 404 returned error can't find the container with id a9f3193eef8632efdaead8741525f576abc311fa2aff08f3d4438ffbe26ad92c
Sep 10 14:57:40.528243 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:40.528186    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"]
Sep 10 14:57:40.538719 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:57:40.538693    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7166542_fbbe_4b3e_9889_3563cad23502.slice/crio-110e7debf29557c82a81fb020c73089cdbe93253754cb936c80415240f41ef30 WatchSource:0}: Error finding container 110e7debf29557c82a81fb020c73089cdbe93253754cb936c80415240f41ef30: Status 404 returned error can't find the container with id 110e7debf29557c82a81fb020c73089cdbe93253754cb936c80415240f41ef30
Sep 10 14:57:41.462586 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.462547    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx" event={"ID":"4bb04e6f-2780-4333-aff1-7e4eefa969ed","Type":"ContainerStarted","Data":"a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344"}
Sep 10 14:57:41.463065 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.462606    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:41.464095 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.464076    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd" event={"ID":"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817","Type":"ContainerStarted","Data":"4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa"}
Sep 10 14:57:41.464158 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.464113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd" event={"ID":"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817","Type":"ContainerStarted","Data":"a9f3193eef8632efdaead8741525f576abc311fa2aff08f3d4438ffbe26ad92c"}
Sep 10 14:57:41.464200 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.464176    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:41.465725 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.465703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv" event={"ID":"c7166542-fbbe-4b3e-9889-3563cad23502","Type":"ContainerStarted","Data":"3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c"}
Sep 10 14:57:41.465847 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.465744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv" event={"ID":"c7166542-fbbe-4b3e-9889-3563cad23502","Type":"ContainerStarted","Data":"110e7debf29557c82a81fb020c73089cdbe93253754cb936c80415240f41ef30"}
Sep 10 14:57:41.465847 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.465769    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:57:41.480570 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.480528    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx" podStartSLOduration=2.094062565 podStartE2EDuration="2.480489838s" totalImagesPullingTime="386.427273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:41.478278368 +0000 UTC m=+931.955338311" watchObservedRunningTime="2026-09-10 14:57:41.480489838 +0000 UTC m=+931.957549781"
Sep 10 14:57:41.493953 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.493904    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd" podStartSLOduration=2.4938853930000002 podStartE2EDuration="2.493885393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:41.493522222 +0000 UTC m=+931.970582170" watchObservedRunningTime="2026-09-10 14:57:41.493885393 +0000 UTC m=+931.970945338"
Sep 10 14:57:41.511133 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:41.511084    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv" podStartSLOduration=2.511066124 podStartE2EDuration="2.511066124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:57:41.507870941 +0000 UTC m=+931.984930885" watchObservedRunningTime="2026-09-10 14:57:41.511066124 +0000 UTC m=+931.988126073"
Sep 10 14:57:47.477337 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:47.477305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:57:49.597523 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:49.597466    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:57:52.474252 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:52.474224    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:57:52.474764 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:57:52.474300    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:58:02.835061 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:02.835027    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:03.601822 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:03.601789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:03.696266 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:03.696233    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:03.987020 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:03.986988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:04.869144 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:04.869113    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:09.763699 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:09.763664    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:09.861793 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:09.861760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:18.264421 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:18.264378    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:19.071798 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:19.071753    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:20.608781 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.608655    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"]
Sep 10 14:58:20.610234 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.609805    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx" podUID="4bb04e6f-2780-4333-aff1-7e4eefa969ed" containerName="maas-api" containerID="cri-o://a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344" gracePeriod=30
Sep 10 14:58:20.636544 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.636493    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:20.669078 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.669046    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"]
Sep 10 14:58:20.669734 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.669461    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd" podUID="5cb5c9fd-dbdb-4574-9d0a-f0546bde3817" containerName="payload-processing" containerID="cri-o://4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa" gracePeriod=30
Sep 10 14:58:20.678294 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.677679    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"]
Sep 10 14:58:20.678294 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.677885    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv" podUID="c7166542-fbbe-4b3e-9889-3563cad23502" containerName="payload-pre-processing" containerID="cri-o://3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c" gracePeriod=30
Sep 10 14:58:20.715369 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.714815    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:20.780449 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.780416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:20.856717 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.856388    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:20.931983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.931852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:20.946622 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.946597    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:58:20.995646 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.995622    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:58:20.999059 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:20.999040    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:58:21.102262 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102231    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-maas-api-tls\") pod \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") "
Sep 10 14:58:21.102459 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k7hr\" (UniqueName: \"kubernetes.io/projected/4bb04e6f-2780-4333-aff1-7e4eefa969ed-kube-api-access-5k7hr\") pod \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") "
Sep 10 14:58:21.102459 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102310    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb04e6f-2780-4333-aff1-7e4eefa969ed-openshift-service-ca\") pod \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") "
Sep 10 14:58:21.102459 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102358    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7166542-fbbe-4b3e-9889-3563cad23502-plugins-config-volume\") pod \"c7166542-fbbe-4b3e-9889-3563cad23502\" (UID: \"c7166542-fbbe-4b3e-9889-3563cad23502\") "
Sep 10 14:58:21.102459 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102415    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-metrics-certs\") pod \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\" (UID: \"4bb04e6f-2780-4333-aff1-7e4eefa969ed\") "
Sep 10 14:58:21.102459 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf52m\" (UniqueName: \"kubernetes.io/projected/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-kube-api-access-tf52m\") pod \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\" (UID: \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\") "
Sep 10 14:58:21.102459 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102457    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84d8z\" (UniqueName: \"kubernetes.io/projected/c7166542-fbbe-4b3e-9889-3563cad23502-kube-api-access-84d8z\") pod \"c7166542-fbbe-4b3e-9889-3563cad23502\" (UID: \"c7166542-fbbe-4b3e-9889-3563cad23502\") "
Sep 10 14:58:21.102997 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102482    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-plugins-config-volume\") pod \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\" (UID: \"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817\") "
Sep 10 14:58:21.102997 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102748    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb04e6f-2780-4333-aff1-7e4eefa969ed-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4bb04e6f-2780-4333-aff1-7e4eefa969ed" (UID: "4bb04e6f-2780-4333-aff1-7e4eefa969ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:58:21.102997 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.102824    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7166542-fbbe-4b3e-9889-3563cad23502-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7166542-fbbe-4b3e-9889-3563cad23502" (UID: "c7166542-fbbe-4b3e-9889-3563cad23502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:58:21.103191 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.103154    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cb5c9fd-dbdb-4574-9d0a-f0546bde3817" (UID: "5cb5c9fd-dbdb-4574-9d0a-f0546bde3817"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:58:21.104903 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.104873    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-kube-api-access-tf52m" (OuterVolumeSpecName: "kube-api-access-tf52m") pod "5cb5c9fd-dbdb-4574-9d0a-f0546bde3817" (UID: "5cb5c9fd-dbdb-4574-9d0a-f0546bde3817"). InnerVolumeSpecName "kube-api-access-tf52m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:58:21.105016 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.104979    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7166542-fbbe-4b3e-9889-3563cad23502-kube-api-access-84d8z" (OuterVolumeSpecName: "kube-api-access-84d8z") pod "c7166542-fbbe-4b3e-9889-3563cad23502" (UID: "c7166542-fbbe-4b3e-9889-3563cad23502"). InnerVolumeSpecName "kube-api-access-84d8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:58:21.105065 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.105027    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4bb04e6f-2780-4333-aff1-7e4eefa969ed" (UID: "4bb04e6f-2780-4333-aff1-7e4eefa969ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:58:21.105101 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.105062    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4bb04e6f-2780-4333-aff1-7e4eefa969ed" (UID: "4bb04e6f-2780-4333-aff1-7e4eefa969ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:58:21.105101 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.105072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb04e6f-2780-4333-aff1-7e4eefa969ed-kube-api-access-5k7hr" (OuterVolumeSpecName: "kube-api-access-5k7hr") pod "4bb04e6f-2780-4333-aff1-7e4eefa969ed" (UID: "4bb04e6f-2780-4333-aff1-7e4eefa969ed"). InnerVolumeSpecName "kube-api-access-5k7hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:58:21.200667 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.200587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:21.203725 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.203705    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:58:21.203800 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.203729    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf52m\" (UniqueName: \"kubernetes.io/projected/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-kube-api-access-tf52m\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:58:21.203800 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.203739    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84d8z\" (UniqueName: \"kubernetes.io/projected/c7166542-fbbe-4b3e-9889-3563cad23502-kube-api-access-84d8z\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:58:21.203800 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.203747    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:58:21.203800 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.203755    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bb04e6f-2780-4333-aff1-7e4eefa969ed-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:58:21.203800 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.203764    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k7hr\" (UniqueName: \"kubernetes.io/projected/4bb04e6f-2780-4333-aff1-7e4eefa969ed-kube-api-access-5k7hr\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:58:21.203800 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.203772    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bb04e6f-2780-4333-aff1-7e4eefa969ed-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:58:21.203800 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.203780    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7166542-fbbe-4b3e-9889-3563cad23502-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:58:21.685211 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.685171    2563 generic.go:356] "Generic (PLEG): container finished" podID="4bb04e6f-2780-4333-aff1-7e4eefa969ed" containerID="a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344" exitCode=0
Sep 10 14:58:21.685697 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.685221    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"
Sep 10 14:58:21.685697 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.685260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx" event={"ID":"4bb04e6f-2780-4333-aff1-7e4eefa969ed","Type":"ContainerDied","Data":"a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344"}
Sep 10 14:58:21.685697 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.685329    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx" event={"ID":"4bb04e6f-2780-4333-aff1-7e4eefa969ed","Type":"ContainerDied","Data":"368bbd6a249d2f91371e190e97a23092842937cf7f93a753fc05fd0967f206dc"}
Sep 10 14:58:21.685697 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.685364    2563 scope.go:117] "RemoveContainer" containerID="a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344"
Sep 10 14:58:21.686713 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.686656    2563 generic.go:356] "Generic (PLEG): container finished" podID="5cb5c9fd-dbdb-4574-9d0a-f0546bde3817" containerID="4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa" exitCode=0
Sep 10 14:58:21.686713 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.686707    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"
Sep 10 14:58:21.686841 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.686702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd" event={"ID":"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817","Type":"ContainerDied","Data":"4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa"}
Sep 10 14:58:21.686841 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.686831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd" event={"ID":"5cb5c9fd-dbdb-4574-9d0a-f0546bde3817","Type":"ContainerDied","Data":"a9f3193eef8632efdaead8741525f576abc311fa2aff08f3d4438ffbe26ad92c"}
Sep 10 14:58:21.688241 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.688217    2563 generic.go:356] "Generic (PLEG): container finished" podID="c7166542-fbbe-4b3e-9889-3563cad23502" containerID="3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c" exitCode=0
Sep 10 14:58:21.688345 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.688298    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"
Sep 10 14:58:21.688345 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.688317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv" event={"ID":"c7166542-fbbe-4b3e-9889-3563cad23502","Type":"ContainerDied","Data":"3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c"}
Sep 10 14:58:21.688457 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.688354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv" event={"ID":"c7166542-fbbe-4b3e-9889-3563cad23502","Type":"ContainerDied","Data":"110e7debf29557c82a81fb020c73089cdbe93253754cb936c80415240f41ef30"}
Sep 10 14:58:21.696766 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.696751    2563 scope.go:117] "RemoveContainer" containerID="a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344"
Sep 10 14:58:21.697035 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:58:21.697015    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344\": container with ID starting with a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344 not found: ID does not exist" containerID="a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344"
Sep 10 14:58:21.697103 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.697043    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344"} err="failed to get container status \"a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344\": rpc error: code = NotFound desc = could not find container \"a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344\": container with ID starting with a323b75caddf0d59ad13a6afad1c95812986e550748b6c9cfea667f3c8c08344 not found: ID does not exist"
Sep 10 14:58:21.697103 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.697060    2563 scope.go:117] "RemoveContainer" containerID="4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa"
Sep 10 14:58:21.705113 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.705097    2563 scope.go:117] "RemoveContainer" containerID="4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa"
Sep 10 14:58:21.705374 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:58:21.705354    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa\": container with ID starting with 4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa not found: ID does not exist" containerID="4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa"
Sep 10 14:58:21.705441 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.705381    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa"} err="failed to get container status \"4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa\": rpc error: code = NotFound desc = could not find container \"4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa\": container with ID starting with 4aec941d5457c2524182f026c8018be25ffdeb8c5d21da002567e7df5a923bfa not found: ID does not exist"
Sep 10 14:58:21.705441 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.705398    2563 scope.go:117] "RemoveContainer" containerID="3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c"
Sep 10 14:58:21.708591 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.708572    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"]
Sep 10 14:58:21.713057 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.713035    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-22282287-7d687d759c-dh8zx"]
Sep 10 14:58:21.715292 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.715277    2563 scope.go:117] "RemoveContainer" containerID="3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c"
Sep 10 14:58:21.715621 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:58:21.715603    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c\": container with ID starting with 3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c not found: ID does not exist" containerID="3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c"
Sep 10 14:58:21.715683 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.715627    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c"} err="failed to get container status \"3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c\": rpc error: code = NotFound desc = could not find container \"3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c\": container with ID starting with 3a8a689181c7677e8d5c7293edccf95d3a7c059205f46f9fc1efc0829ae0512c not found: ID does not exist"
Sep 10 14:58:21.722560 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.722539    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"]
Sep 10 14:58:21.726322 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.726300    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-22282287-6df7d8c677-btjmd"]
Sep 10 14:58:21.735045 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.735025    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"]
Sep 10 14:58:21.738889 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:21.738870    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-22282287-86b8764c58-mqhhv"]
Sep 10 14:58:22.039501 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:22.039471    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb04e6f-2780-4333-aff1-7e4eefa969ed" path="/var/lib/kubelet/pods/4bb04e6f-2780-4333-aff1-7e4eefa969ed/volumes"
Sep 10 14:58:22.039935 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:22.039918    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cb5c9fd-dbdb-4574-9d0a-f0546bde3817" path="/var/lib/kubelet/pods/5cb5c9fd-dbdb-4574-9d0a-f0546bde3817/volumes"
Sep 10 14:58:22.040248 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:22.040236    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7166542-fbbe-4b3e-9889-3563cad23502" path="/var/lib/kubelet/pods/c7166542-fbbe-4b3e-9889-3563cad23502/volumes"
Sep 10 14:58:27.073795 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:27.073749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:27.184830 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:27.184796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:27.378896 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:27.378804    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:27.710707 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:27.710666    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:27.809029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:27.809002    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:27.920702 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:27.920665    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:33.515090 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:33.514988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:33.620725 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:33.620690    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:33.715410 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:33.715363    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:33.886788 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:33.886703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:39.704581 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:39.704547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:39.804893 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:39.804860    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:40.670000 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:40.669965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:40.994345 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:40.994267    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:41.328764 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:41.328689    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:44.277873 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:44.277833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:44.366126 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:44.366089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:45.466269 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.466234    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"]
Sep 10 14:58:45.466863 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.466843    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb5c9fd-dbdb-4574-9d0a-f0546bde3817" containerName="payload-processing"
Sep 10 14:58:45.466910 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.466868    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb5c9fd-dbdb-4574-9d0a-f0546bde3817" containerName="payload-processing"
Sep 10 14:58:45.466949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.466914    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7166542-fbbe-4b3e-9889-3563cad23502" containerName="payload-pre-processing"
Sep 10 14:58:45.466949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.466923    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7166542-fbbe-4b3e-9889-3563cad23502" containerName="payload-pre-processing"
Sep 10 14:58:45.466949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.466933    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb04e6f-2780-4333-aff1-7e4eefa969ed" containerName="maas-api"
Sep 10 14:58:45.466949 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.466942    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb04e6f-2780-4333-aff1-7e4eefa969ed" containerName="maas-api"
Sep 10 14:58:45.467078 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.467047    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bb04e6f-2780-4333-aff1-7e4eefa969ed" containerName="maas-api"
Sep 10 14:58:45.467078 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.467061    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7166542-fbbe-4b3e-9889-3563cad23502" containerName="payload-pre-processing"
Sep 10 14:58:45.467078 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.467074    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cb5c9fd-dbdb-4574-9d0a-f0546bde3817" containerName="payload-processing"
Sep 10 14:58:45.470738 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.470719    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.472885 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.472864    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5fb994-serving-cert\""
Sep 10 14:58:45.476690 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.476667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"]
Sep 10 14:58:45.502879 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.502848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:45.517886 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.517855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.518012 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.517922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.518012 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.517991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a704f10-6b7d-489c-a890-82daa2a4be52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.518084 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.518040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62wl4\" (UniqueName: \"kubernetes.io/projected/7a704f10-6b7d-489c-a890-82daa2a4be52-kube-api-access-62wl4\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.562746 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.562703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:45.619137 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.619112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62wl4\" (UniqueName: \"kubernetes.io/projected/7a704f10-6b7d-489c-a890-82daa2a4be52-kube-api-access-62wl4\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.619292 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.619165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.619292 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:58:45.619259    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-serving-cert: secret "maas-api-e2e-shared-a-w2-5fb994-serving-cert" not found
Sep 10 14:58:45.619374 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:58:45.619324    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-maas-api-tls podName:7a704f10-6b7d-489c-a890-82daa2a4be52 nodeName:}" failed. No retries permitted until 2026-09-10 14:58:46.119308908 +0000 UTC m=+996.596368830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-maas-api-tls") pod "maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk" (UID: "7a704f10-6b7d-489c-a890-82daa2a4be52") : secret "maas-api-e2e-shared-a-w2-5fb994-serving-cert" not found
Sep 10 14:58:45.619374 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.619336    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.619491 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.619373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a704f10-6b7d-489c-a890-82daa2a4be52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.619954 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.619933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a704f10-6b7d-489c-a890-82daa2a4be52-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.621835 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.621813    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.627406 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.627373    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62wl4\" (UniqueName: \"kubernetes.io/projected/7a704f10-6b7d-489c-a890-82daa2a4be52-kube-api-access-62wl4\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:45.861952 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:45.861831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:46.061695 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.061662    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:46.109370 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.109328    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"]
Sep 10 14:58:46.113804 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.113735    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:58:46.115755 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.115714    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5fb994-dockercfg-2p2rf\""
Sep 10 14:58:46.116009 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.115960    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5fb994\""
Sep 10 14:58:46.123026 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.123001    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"]
Sep 10 14:58:46.125770 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.125748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:46.128205 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.128180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:46.137273 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.137248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:46.146487 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.146468    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"]
Sep 10 14:58:46.149787 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.149773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:46.160172 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.160151    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"]
Sep 10 14:58:46.226706 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.226673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq5kv\" (UniqueName: \"kubernetes.io/projected/979d03d3-e3c3-4f22-bfd9-8811e22c2734-kube-api-access-jq5kv\") pod \"payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg\" (UID: \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:58:46.226888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.226793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2jp\" (UniqueName: \"kubernetes.io/projected/8ba2dd72-5250-4fba-b733-71e3f4798e09-kube-api-access-tc2jp\") pod \"payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z\" (UID: \"8ba2dd72-5250-4fba-b733-71e3f4798e09\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:46.226888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.226842    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979d03d3-e3c3-4f22-bfd9-8811e22c2734-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg\" (UID: \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:58:46.227080 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.227046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba2dd72-5250-4fba-b733-71e3f4798e09-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z\" (UID: \"8ba2dd72-5250-4fba-b733-71e3f4798e09\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:46.328343 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.328299    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba2dd72-5250-4fba-b733-71e3f4798e09-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z\" (UID: \"8ba2dd72-5250-4fba-b733-71e3f4798e09\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:46.328576 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.328394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq5kv\" (UniqueName: \"kubernetes.io/projected/979d03d3-e3c3-4f22-bfd9-8811e22c2734-kube-api-access-jq5kv\") pod \"payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg\" (UID: \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:58:46.328576 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.328481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2jp\" (UniqueName: \"kubernetes.io/projected/8ba2dd72-5250-4fba-b733-71e3f4798e09-kube-api-access-tc2jp\") pod \"payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z\" (UID: \"8ba2dd72-5250-4fba-b733-71e3f4798e09\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:46.328576 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.328531    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979d03d3-e3c3-4f22-bfd9-8811e22c2734-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg\" (UID: \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:58:46.329143 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.329117    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba2dd72-5250-4fba-b733-71e3f4798e09-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z\" (UID: \"8ba2dd72-5250-4fba-b733-71e3f4798e09\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:46.329806 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.329787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979d03d3-e3c3-4f22-bfd9-8811e22c2734-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg\" (UID: \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:58:46.336392 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.336363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2jp\" (UniqueName: \"kubernetes.io/projected/8ba2dd72-5250-4fba-b733-71e3f4798e09-kube-api-access-tc2jp\") pod \"payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z\" (UID: \"8ba2dd72-5250-4fba-b733-71e3f4798e09\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:46.337058 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.337032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq5kv\" (UniqueName: \"kubernetes.io/projected/979d03d3-e3c3-4f22-bfd9-8811e22c2734-kube-api-access-jq5kv\") pod \"payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg\" (UID: \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:58:46.384461 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.384386    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:46.426229 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.425852    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:58:46.460163 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.460137    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:46.656468 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.656444    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"]
Sep 10 14:58:46.658268 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:58:46.658243    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a704f10_6b7d_489c_a890_82daa2a4be52.slice/crio-612c378a3213b32a4c3bb193c9b8c245e95986bc90410e3256f9c2ce4386f6aa WatchSource:0}: Error finding container 612c378a3213b32a4c3bb193c9b8c245e95986bc90410e3256f9c2ce4386f6aa: Status 404 returned error can't find the container with id 612c378a3213b32a4c3bb193c9b8c245e95986bc90410e3256f9c2ce4386f6aa
Sep 10 14:58:46.708312 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.708290    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"]
Sep 10 14:58:46.711427 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:58:46.711392    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979d03d3_e3c3_4f22_bfd9_8811e22c2734.slice/crio-5b66208a51f09c9b11d82d95d56db1fa5dcef262bf7d389a1d5e1523bf88b951 WatchSource:0}: Error finding container 5b66208a51f09c9b11d82d95d56db1fa5dcef262bf7d389a1d5e1523bf88b951: Status 404 returned error can't find the container with id 5b66208a51f09c9b11d82d95d56db1fa5dcef262bf7d389a1d5e1523bf88b951
Sep 10 14:58:46.744149 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.744129    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"]
Sep 10 14:58:46.750678 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:58:46.750651    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba2dd72_5250_4fba_b733_71e3f4798e09.slice/crio-b08813bb8d7d95c437bab88cca2a0934c5dd3149deb6b51ab1c81ecb3177145e WatchSource:0}: Error finding container b08813bb8d7d95c437bab88cca2a0934c5dd3149deb6b51ab1c81ecb3177145e: Status 404 returned error can't find the container with id b08813bb8d7d95c437bab88cca2a0934c5dd3149deb6b51ab1c81ecb3177145e
Sep 10 14:58:46.822097 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.822059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z" event={"ID":"8ba2dd72-5250-4fba-b733-71e3f4798e09","Type":"ContainerStarted","Data":"f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e"}
Sep 10 14:58:46.822283 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.822124    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z" event={"ID":"8ba2dd72-5250-4fba-b733-71e3f4798e09","Type":"ContainerStarted","Data":"b08813bb8d7d95c437bab88cca2a0934c5dd3149deb6b51ab1c81ecb3177145e"}
Sep 10 14:58:46.822283 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.822222    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:46.823889 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.823863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk" event={"ID":"7a704f10-6b7d-489c-a890-82daa2a4be52","Type":"ContainerStarted","Data":"612c378a3213b32a4c3bb193c9b8c245e95986bc90410e3256f9c2ce4386f6aa"}
Sep 10 14:58:46.825875 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.825805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg" event={"ID":"979d03d3-e3c3-4f22-bfd9-8811e22c2734","Type":"ContainerStarted","Data":"f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce"}
Sep 10 14:58:46.825875 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.825857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg" event={"ID":"979d03d3-e3c3-4f22-bfd9-8811e22c2734","Type":"ContainerStarted","Data":"5b66208a51f09c9b11d82d95d56db1fa5dcef262bf7d389a1d5e1523bf88b951"}
Sep 10 14:58:46.826104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.826086    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:58:46.839204 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.839161    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z" podStartSLOduration=0.839147976 podStartE2EDuration="839.147976ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:58:46.838061573 +0000 UTC m=+997.315121519" watchObservedRunningTime="2026-09-10 14:58:46.839147976 +0000 UTC m=+997.316207960"
Sep 10 14:58:46.852870 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:46.852803    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg" podStartSLOduration=0.852789275 podStartE2EDuration="852.789275ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:58:46.852286175 +0000 UTC m=+997.329346120" watchObservedRunningTime="2026-09-10 14:58:46.852789275 +0000 UTC m=+997.329849219"
Sep 10 14:58:47.832433 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:47.832394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk" event={"ID":"7a704f10-6b7d-489c-a890-82daa2a4be52","Type":"ContainerStarted","Data":"9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f"}
Sep 10 14:58:47.832842 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:47.832647    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:47.849744 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:47.849689    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk" podStartSLOduration=2.385831106 podStartE2EDuration="2.849674392s" totalImagesPullingTime="463.843286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:58:47.848883553 +0000 UTC m=+998.325943521" watchObservedRunningTime="2026-09-10 14:58:47.849674392 +0000 UTC m=+998.326734567"
Sep 10 14:58:53.795271 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:53.795233    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"]
Sep 10 14:58:53.816194 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:53.816160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:53.843677 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:53.843651    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 14:58:56.079546 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:56.079497    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:56.414301 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:56.414268    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:58:57.834449 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:57.834417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 14:58:57.835187 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:58:57.835169    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 14:59:06.915917 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:06.915883    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:07.502916 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.502884    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"]
Sep 10 14:59:07.503139 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.503092    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5" podUID="0905ed37-e70e-4ebd-acf4-3602153bc2eb" containerName="maas-api" containerID="cri-o://9acbda0b049967776f62323ac6659bda7171fd939a7c4becec39c60354b3daf5" gracePeriod=30
Sep 10 14:59:07.537691 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.537636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:07.557881 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.557830    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"]
Sep 10 14:59:07.558099 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.558072    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz" podUID="fd5aaa67-abde-4b8b-b901-39de7daf0ea0" containerName="payload-processing" containerID="cri-o://72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759" gracePeriod=30
Sep 10 14:59:07.638386 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.638351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:07.738763 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.738730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:07.839201 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.839164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:07.853581 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.853485    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:59:07.935601 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.935571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqxp7\" (UniqueName: \"kubernetes.io/projected/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-kube-api-access-jqxp7\") pod \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\" (UID: \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\") "
Sep 10 14:59:07.935601 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.935600    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-plugins-config-volume\") pod \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\" (UID: \"fd5aaa67-abde-4b8b-b901-39de7daf0ea0\") "
Sep 10 14:59:07.936066 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.935978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd5aaa67-abde-4b8b-b901-39de7daf0ea0" (UID: "fd5aaa67-abde-4b8b-b901-39de7daf0ea0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:59:07.938662 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.938633    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-kube-api-access-jqxp7" (OuterVolumeSpecName: "kube-api-access-jqxp7") pod "fd5aaa67-abde-4b8b-b901-39de7daf0ea0" (UID: "fd5aaa67-abde-4b8b-b901-39de7daf0ea0"). InnerVolumeSpecName "kube-api-access-jqxp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:59:07.942834 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.942804    2563 generic.go:356] "Generic (PLEG): container finished" podID="fd5aaa67-abde-4b8b-b901-39de7daf0ea0" containerID="72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759" exitCode=0
Sep 10 14:59:07.942971 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.942841    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"
Sep 10 14:59:07.942971 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.942883    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz" event={"ID":"fd5aaa67-abde-4b8b-b901-39de7daf0ea0","Type":"ContainerDied","Data":"72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759"}
Sep 10 14:59:07.942971 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.942941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz" event={"ID":"fd5aaa67-abde-4b8b-b901-39de7daf0ea0","Type":"ContainerDied","Data":"8ac032acff8c78af0a923afceffe144363037d4ac7d14a9fbe58ee0459597c58"}
Sep 10 14:59:07.942971 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.942971    2563 scope.go:117] "RemoveContainer" containerID="72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759"
Sep 10 14:59:07.945326 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.945198    2563 generic.go:356] "Generic (PLEG): container finished" podID="0905ed37-e70e-4ebd-acf4-3602153bc2eb" containerID="9acbda0b049967776f62323ac6659bda7171fd939a7c4becec39c60354b3daf5" exitCode=0
Sep 10 14:59:07.945326 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.945260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5" event={"ID":"0905ed37-e70e-4ebd-acf4-3602153bc2eb","Type":"ContainerDied","Data":"9acbda0b049967776f62323ac6659bda7171fd939a7c4becec39c60354b3daf5"}
Sep 10 14:59:07.953391 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.953371    2563 scope.go:117] "RemoveContainer" containerID="72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759"
Sep 10 14:59:07.953758 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:59:07.953738    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759\": container with ID starting with 72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759 not found: ID does not exist" containerID="72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759"
Sep 10 14:59:07.953836 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.953769    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759"} err="failed to get container status \"72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759\": rpc error: code = NotFound desc = could not find container \"72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759\": container with ID starting with 72b1bf600576bff18f383a78cb4eafea5f69f04abfa4e2bd081d26c1693cf759 not found: ID does not exist"
Sep 10 14:59:07.969179 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.969155    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"]
Sep 10 14:59:07.973491 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.973474    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:59:07.975884 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:07.975865    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-74eee9-798d95f9bc-74cqz"]
Sep 10 14:59:08.033189 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.033148    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5aaa67-abde-4b8b-b901-39de7daf0ea0" path="/var/lib/kubelet/pods/fd5aaa67-abde-4b8b-b901-39de7daf0ea0/volumes"
Sep 10 14:59:08.036407 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.036346    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqxp7\" (UniqueName: \"kubernetes.io/projected/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-kube-api-access-jqxp7\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:08.036407 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.036372    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5aaa67-abde-4b8b-b901-39de7daf0ea0-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:08.137573 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.137547    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0905ed37-e70e-4ebd-acf4-3602153bc2eb-openshift-service-ca\") pod \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") "
Sep 10 14:59:08.137734 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.137599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-maas-api-tls\") pod \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") "
Sep 10 14:59:08.137734 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.137659    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-metrics-certs\") pod \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") "
Sep 10 14:59:08.137734 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.137725    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dn9t\" (UniqueName: \"kubernetes.io/projected/0905ed37-e70e-4ebd-acf4-3602153bc2eb-kube-api-access-6dn9t\") pod \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\" (UID: \"0905ed37-e70e-4ebd-acf4-3602153bc2eb\") "
Sep 10 14:59:08.137955 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.137934    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0905ed37-e70e-4ebd-acf4-3602153bc2eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0905ed37-e70e-4ebd-acf4-3602153bc2eb" (UID: "0905ed37-e70e-4ebd-acf4-3602153bc2eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:59:08.138232 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.138204    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0905ed37-e70e-4ebd-acf4-3602153bc2eb-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:08.139925 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.139901    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0905ed37-e70e-4ebd-acf4-3602153bc2eb-kube-api-access-6dn9t" (OuterVolumeSpecName: "kube-api-access-6dn9t") pod "0905ed37-e70e-4ebd-acf4-3602153bc2eb" (UID: "0905ed37-e70e-4ebd-acf4-3602153bc2eb"). InnerVolumeSpecName "kube-api-access-6dn9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:59:08.139925 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.139916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0905ed37-e70e-4ebd-acf4-3602153bc2eb" (UID: "0905ed37-e70e-4ebd-acf4-3602153bc2eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:59:08.140047 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.139902    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0905ed37-e70e-4ebd-acf4-3602153bc2eb" (UID: "0905ed37-e70e-4ebd-acf4-3602153bc2eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:59:08.147253 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.147226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:08.239227 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.239195    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:08.239227 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.239221    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0905ed37-e70e-4ebd-acf4-3602153bc2eb-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:08.239227 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.239231    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dn9t\" (UniqueName: \"kubernetes.io/projected/0905ed37-e70e-4ebd-acf4-3602153bc2eb-kube-api-access-6dn9t\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:08.816467 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.816403    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws" podUID="c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" containerName="main" containerID="cri-o://92bd66e7581ccd87e079311a9b7728c1aa24c01cd37a1ceb74d042e7b15f131b" gracePeriod=45
Sep 10 14:59:08.951576 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.951543    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"
Sep 10 14:59:08.951576 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.951551    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5" event={"ID":"0905ed37-e70e-4ebd-acf4-3602153bc2eb","Type":"ContainerDied","Data":"035de8470216d20ad40b34e15c211f218158b5e99566c394819af26a9334143f"}
Sep 10 14:59:08.952067 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.951619    2563 scope.go:117] "RemoveContainer" containerID="9acbda0b049967776f62323ac6659bda7171fd939a7c4becec39c60354b3daf5"
Sep 10 14:59:08.954953 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.954924    2563 generic.go:356] "Generic (PLEG): container finished" podID="c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" containerID="92bd66e7581ccd87e079311a9b7728c1aa24c01cd37a1ceb74d042e7b15f131b" exitCode=0
Sep 10 14:59:08.955089 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.954960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws" event={"ID":"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1","Type":"ContainerDied","Data":"92bd66e7581ccd87e079311a9b7728c1aa24c01cd37a1ceb74d042e7b15f131b"}
Sep 10 14:59:08.975466 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.975441    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"]
Sep 10 14:59:08.979604 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:08.979577    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-74eee9-65cdd6b945-dt2t5"]
Sep 10 14:59:09.078686 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.078660    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:59:09.247566 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.247495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-model-cache\") pod \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") "
Sep 10 14:59:09.247780 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.247585    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9t92\" (UniqueName: \"kubernetes.io/projected/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-kube-api-access-k9t92\") pod \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") "
Sep 10 14:59:09.247780 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.247620    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-home\") pod \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") "
Sep 10 14:59:09.247780 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.247674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-dshm\") pod \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") "
Sep 10 14:59:09.247780 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.247703    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tmp-dir\") pod \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") "
Sep 10 14:59:09.247780 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.247739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tls-certs\") pod \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\" (UID: \"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1\") "
Sep 10 14:59:09.248039 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.247834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" (UID: "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:59:09.248039 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.247945    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-home" (OuterVolumeSpecName: "home") pod "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" (UID: "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:59:09.248039 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.247991    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" (UID: "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:59:09.248159 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.248079    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-model-cache\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:09.248159 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.248095    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-home\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:09.248159 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.248104    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tmp-dir\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:09.249982 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.249959    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" (UID: "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:59:09.250068 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.249966    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-dshm" (OuterVolumeSpecName: "dshm") pod "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" (UID: "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 14:59:09.250068 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.250027    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-kube-api-access-k9t92" (OuterVolumeSpecName: "kube-api-access-k9t92") pod "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" (UID: "c8c093a2-5eed-44f3-9a72-6a0e9867a7a1"). InnerVolumeSpecName "kube-api-access-k9t92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:59:09.320528 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.320459    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:09.348566 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.348537    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9t92\" (UniqueName: \"kubernetes.io/projected/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-kube-api-access-k9t92\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:09.348566 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.348562    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-dshm\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:09.348566 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.348571    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1-tls-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:09.665921 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.665882    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:09.960112 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.959959    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"]
Sep 10 14:59:09.960695 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.960632    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5aaa67-abde-4b8b-b901-39de7daf0ea0" containerName="payload-processing"
Sep 10 14:59:09.960695 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.960654    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5aaa67-abde-4b8b-b901-39de7daf0ea0" containerName="payload-processing"
Sep 10 14:59:09.960695 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.960682    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" containerName="main"
Sep 10 14:59:09.960695 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.960690    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" containerName="main"
Sep 10 14:59:09.960695 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.960713    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0905ed37-e70e-4ebd-acf4-3602153bc2eb" containerName="maas-api"
Sep 10 14:59:09.961029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.960737    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0905ed37-e70e-4ebd-acf4-3602153bc2eb" containerName="maas-api"
Sep 10 14:59:09.961029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.960838    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" containerName="main"
Sep 10 14:59:09.961029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.960852    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0905ed37-e70e-4ebd-acf4-3602153bc2eb" containerName="maas-api"
Sep 10 14:59:09.961029 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.960866    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5aaa67-abde-4b8b-b901-39de7daf0ea0" containerName="payload-processing"
Sep 10 14:59:09.964598 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.964576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:09.965157 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.965132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws" event={"ID":"c8c093a2-5eed-44f3-9a72-6a0e9867a7a1","Type":"ContainerDied","Data":"eb0f767253f9006c0dd5e536a9bc299d5578c82e83434b990fccbf376018895d"}
Sep 10 14:59:09.965231 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.965172    2563 scope.go:117] "RemoveContainer" containerID="92bd66e7581ccd87e079311a9b7728c1aa24c01cd37a1ceb74d042e7b15f131b"
Sep 10 14:59:09.965231 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.965175    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"
Sep 10 14:59:09.966703 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.966686    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 14:59:09.966809 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.966686    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5899e8-openshift-default-dockercfg-5gzn9\""
Sep 10 14:59:09.976570 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.976459    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"]
Sep 10 14:59:09.991095 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:09.991062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:10.007893 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.007867    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"]
Sep 10 14:59:10.010903 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.010876    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-74eee9/test-model-74eee9-kserve-55f469dcbc-pvpws"]
Sep 10 14:59:10.032171 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.032148    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0905ed37-e70e-4ebd-acf4-3602153bc2eb" path="/var/lib/kubelet/pods/0905ed37-e70e-4ebd-acf4-3602153bc2eb/volumes"
Sep 10 14:59:10.032584 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.032570    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c093a2-5eed-44f3-9a72-6a0e9867a7a1" path="/var/lib/kubelet/pods/c8c093a2-5eed-44f3-9a72-6a0e9867a7a1/volumes"
Sep 10 14:59:10.054383 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.054359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4tx\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-kube-api-access-9x4tx\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.054501 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.054387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-data\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.054501 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.054419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-token\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.054501 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.054481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-credential-socket\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.054659 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.054532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-certs\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.054659 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.054553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-envoy\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.054659 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.054578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-podinfo\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.054659 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.054613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istiod-ca-cert\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.054797 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.054660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-socket\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.091628 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.091590    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:10.155332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-token\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155457 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-credential-socket\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155457 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-certs\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155457 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155394    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-envoy\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155457 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155418    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-podinfo\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155457 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istiod-ca-cert\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155747 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-socket\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155747 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4tx\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-kube-api-access-9x4tx\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155747 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-data\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155903 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155785    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-certs\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.155968 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.155939    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-credential-socket\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.156051 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.156026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-data\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.156166 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.156074    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-socket\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.156272 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.156259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istiod-ca-cert\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.157839 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.157822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-envoy\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.157962 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.157947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-podinfo\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.162882 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.162857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4tx\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-kube-api-access-9x4tx\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.163027 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.163000    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-token\") pod \"e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") " pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.279859 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.279778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:10.558747 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.558720    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"]
Sep 10 14:59:10.560632 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:59:10.560607    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0c2c19_7193_44f8_b1fc_d3c7bc644b6f.slice/crio-42e1a2ac9e942f7ffb20cfaca6adb638d02c18b31812b797b58dc8a62eb49c3f WatchSource:0}: Error finding container 42e1a2ac9e942f7ffb20cfaca6adb638d02c18b31812b797b58dc8a62eb49c3f: Status 404 returned error can't find the container with id 42e1a2ac9e942f7ffb20cfaca6adb638d02c18b31812b797b58dc8a62eb49c3f
Sep 10 14:59:10.562897 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.562872    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:59:10.562963 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.562929    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:59:10.563000 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.562961    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:59:10.973239 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.973199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" event={"ID":"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f","Type":"ContainerStarted","Data":"f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf"}
Sep 10 14:59:10.973650 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.973260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" event={"ID":"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f","Type":"ContainerStarted","Data":"42e1a2ac9e942f7ffb20cfaca6adb638d02c18b31812b797b58dc8a62eb49c3f"}
Sep 10 14:59:10.993957 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:10.993901    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" podStartSLOduration=1.993882339 podStartE2EDuration="1.993882339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:10.992917057 +0000 UTC m=+1021.469977014" watchObservedRunningTime="2026-09-10 14:59:10.993882339 +0000 UTC m=+1021.470942284"
Sep 10 14:59:11.280818 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:11.280740    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:11.285382 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:11.285361    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:11.979681 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:11.979648    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:11.980617 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:11.980598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:13.014496 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:13.014393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:13.115978 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:13.115933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:14.281429 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.281393    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"]
Sep 10 14:59:14.286054 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.286030    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.287872 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.287845    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-5899e8-serving-cert\""
Sep 10 14:59:14.294472 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.294446    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"]
Sep 10 14:59:14.320083 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.320042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:14.399372 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.399337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.399574 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.399379    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.399574 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.399399    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvpv\" (UniqueName: \"kubernetes.io/projected/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-kube-api-access-jgvpv\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.399657 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.399582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.420143 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.420109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:14.485283 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.485236    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:14.500446 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.500416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.500620 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.500460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.500620 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.500558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.500620 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.500581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvpv\" (UniqueName: \"kubernetes.io/projected/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-kube-api-access-jgvpv\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.501249 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.501231    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.503044 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.503012    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.503134 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.503058    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.508447 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.508423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvpv\" (UniqueName: \"kubernetes.io/projected/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-kube-api-access-jgvpv\") pod \"maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.598860 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.598767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:14.689853 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.689825    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:14.799534 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.799488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:14.903871 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.903844    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"]
Sep 10 14:59:14.905355 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:59:14.905328    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e70000_fc43_4ad5_b7f7_75c735a2b9c3.slice/crio-2328f24baf7f3bbadbbf725fda2550cd2b978f471d5562222fbfe99a8264450c WatchSource:0}: Error finding container 2328f24baf7f3bbadbbf725fda2550cd2b978f471d5562222fbfe99a8264450c: Status 404 returned error can't find the container with id 2328f24baf7f3bbadbbf725fda2550cd2b978f471d5562222fbfe99a8264450c
Sep 10 14:59:14.938872 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.938839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:14.997147 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:14.997111    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69" event={"ID":"32e70000-fc43-4ad5-b7f7-75c735a2b9c3","Type":"ContainerStarted","Data":"2328f24baf7f3bbadbbf725fda2550cd2b978f471d5562222fbfe99a8264450c"}
Sep 10 14:59:15.038582 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:15.038534    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:15.990105 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:15.990070    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:16.005115 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:16.005078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69" event={"ID":"32e70000-fc43-4ad5-b7f7-75c735a2b9c3","Type":"ContainerStarted","Data":"c490608b2ee4cba5713933ea68cc6a4629e7007c7687b235b8c5218cb0b378fd"}
Sep 10 14:59:16.005292 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:16.005200    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:16.026241 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:16.026192    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69" podStartSLOduration=1.6154327560000001 podStartE2EDuration="2.026174539s" totalImagesPullingTime="410.741783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:16.023578396 +0000 UTC m=+1026.500638346" watchObservedRunningTime="2026-09-10 14:59:16.026174539 +0000 UTC m=+1026.503234522"
Sep 10 14:59:16.734468 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:16.734418    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:22.021436 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:22.021397    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 14:59:25.528691 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:25.528651    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:26.158933 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.158891    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"]
Sep 10 14:59:26.159139 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.159102    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b" podUID="d65db817-a615-4746-8441-1f345f37ee5c" containerName="maas-api" containerID="cri-o://386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c" gracePeriod=30
Sep 10 14:59:26.195000 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.194973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:26.288193 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.288161    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:26.350495 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.350352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:26.421796 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.421724    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:59:26.422840 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.422811    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:26.490975 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.490939    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:26.512776 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.512741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-metrics-certs\") pod \"d65db817-a615-4746-8441-1f345f37ee5c\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") "
Sep 10 14:59:26.512928 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.512794    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-maas-api-tls\") pod \"d65db817-a615-4746-8441-1f345f37ee5c\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") "
Sep 10 14:59:26.512928 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.512834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65db817-a615-4746-8441-1f345f37ee5c-openshift-service-ca\") pod \"d65db817-a615-4746-8441-1f345f37ee5c\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") "
Sep 10 14:59:26.512928 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.512910    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v24ql\" (UniqueName: \"kubernetes.io/projected/d65db817-a615-4746-8441-1f345f37ee5c-kube-api-access-v24ql\") pod \"d65db817-a615-4746-8441-1f345f37ee5c\" (UID: \"d65db817-a615-4746-8441-1f345f37ee5c\") "
Sep 10 14:59:26.513331 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.513282    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65db817-a615-4746-8441-1f345f37ee5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d65db817-a615-4746-8441-1f345f37ee5c" (UID: "d65db817-a615-4746-8441-1f345f37ee5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 14:59:26.514954 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.514928    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d65db817-a615-4746-8441-1f345f37ee5c" (UID: "d65db817-a615-4746-8441-1f345f37ee5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:59:26.515063 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.515033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d65db817-a615-4746-8441-1f345f37ee5c" (UID: "d65db817-a615-4746-8441-1f345f37ee5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 14:59:26.515387 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.515371    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65db817-a615-4746-8441-1f345f37ee5c-kube-api-access-v24ql" (OuterVolumeSpecName: "kube-api-access-v24ql") pod "d65db817-a615-4746-8441-1f345f37ee5c" (UID: "d65db817-a615-4746-8441-1f345f37ee5c"). InnerVolumeSpecName "kube-api-access-v24ql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 14:59:26.613911 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.613870    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:26.613911 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.613909    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d65db817-a615-4746-8441-1f345f37ee5c-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:26.614300 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.613924    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d65db817-a615-4746-8441-1f345f37ee5c-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:26.614300 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.613939    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v24ql\" (UniqueName: \"kubernetes.io/projected/d65db817-a615-4746-8441-1f345f37ee5c-kube-api-access-v24ql\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 14:59:26.821495 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:26.821407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:27.068010 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:27.067969    2563 generic.go:356] "Generic (PLEG): container finished" podID="d65db817-a615-4746-8441-1f345f37ee5c" containerID="386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c" exitCode=0
Sep 10 14:59:27.068010 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:27.068012    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"
Sep 10 14:59:27.068257 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:27.068050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b" event={"ID":"d65db817-a615-4746-8441-1f345f37ee5c","Type":"ContainerDied","Data":"386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c"}
Sep 10 14:59:27.068257 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:27.068101    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b" event={"ID":"d65db817-a615-4746-8441-1f345f37ee5c","Type":"ContainerDied","Data":"99f211a288cc73e59ccd7d23ef124266f441dcfd939133d3888f91209b495977"}
Sep 10 14:59:27.068257 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:27.068122    2563 scope.go:117] "RemoveContainer" containerID="386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c"
Sep 10 14:59:27.078888 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:27.078870    2563 scope.go:117] "RemoveContainer" containerID="386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c"
Sep 10 14:59:27.079225 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:59:27.079198    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c\": container with ID starting with 386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c not found: ID does not exist" containerID="386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c"
Sep 10 14:59:27.079340 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:27.079232    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c"} err="failed to get container status \"386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c\": rpc error: code = NotFound desc = could not find container \"386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c\": container with ID starting with 386a4ddb46e31bba06fec15f1c605b96aed82a49bf8c3e37f54232e82320366c not found: ID does not exist"
Sep 10 14:59:27.090365 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:27.090340    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"]
Sep 10 14:59:27.094770 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:27.094746    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ff661a-864dc9f8c5-g466b"]
Sep 10 14:59:28.021671 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:28.021625    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 14:59:28.022082 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:28.021686    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 14:59:28.032943 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:28.032919    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d65db817-a615-4746-8441-1f345f37ee5c" path="/var/lib/kubelet/pods/d65db817-a615-4746-8441-1f345f37ee5c/volumes"
Sep 10 14:59:30.085519 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:30.085477    2563 generic.go:356] "Generic (PLEG): container finished" podID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerID="f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf" exitCode=0
Sep 10 14:59:30.085950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:30.085552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" event={"ID":"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f","Type":"ContainerDied","Data":"f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf"}
Sep 10 14:59:30.085950 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:30.085930    2563 scope.go:117] "RemoveContainer" containerID="f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf"
Sep 10 14:59:30.087012 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:30.086986    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:59:30.087073 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:30.087058    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:59:30.087109 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:30.087094    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 14:59:31.092338 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:31.092303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" event={"ID":"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f","Type":"ContainerStarted","Data":"cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5"}
Sep 10 14:59:31.280364 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:31.280330    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:31.280656 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:31.280636    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 10 14:59:31.280716 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:31.280697    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 10 14:59:32.280967 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:32.280928    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 10 14:59:32.281348 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:32.280990    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 10 14:59:32.834341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:32.834306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:32.936214 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:32.936178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:33.120417 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:33.120325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:33.280268 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:33.280233    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 10 14:59:33.280435 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:33.280295    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 10 14:59:33.673150 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:33.673120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:33.997076 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:33.996996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:34.280332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:34.280250    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 10 14:59:34.280332 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:34.280317    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 10 14:59:34.377101 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:34.377067    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:35.281013 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:35.280977    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 10 14:59:35.281414 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:35.281040    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 10 14:59:35.543943 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:35.543846    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:36.396063 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:36.396035    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:36.396474 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:36.396305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:36.396994 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:36.396977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 14:59:36.580489 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:36.580451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:36.679341 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:36.679260    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:37.692324 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.692289    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"]
Sep 10 14:59:37.692827 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.692748    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d65db817-a615-4746-8441-1f345f37ee5c" containerName="maas-api"
Sep 10 14:59:37.692827 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.692762    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d65db817-a615-4746-8441-1f345f37ee5c" containerName="maas-api"
Sep 10 14:59:37.692910 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.692836    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d65db817-a615-4746-8441-1f345f37ee5c" containerName="maas-api"
Sep 10 14:59:37.695832 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.695815    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.697983 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.697957    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-16ad45-serving-cert\""
Sep 10 14:59:37.706455 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.706426    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"]
Sep 10 14:59:37.737165 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.737132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:37.823662 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.823626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.823877 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.823692    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.823877 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.823787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.823991 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.823897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjhr\" (UniqueName: \"kubernetes.io/projected/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-kube-api-access-5vjhr\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.835125 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.835098    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:37.924682 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.924643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.924882 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.924691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.924882 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.924723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.924882 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.924770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjhr\" (UniqueName: \"kubernetes.io/projected/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-kube-api-access-5vjhr\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.924882 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:59:37.924801    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-serving-cert: secret "maas-api-e2e-shared-a-w3-16ad45-serving-cert" not found
Sep 10 14:59:37.924882 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:59:37.924861    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-maas-api-tls podName:5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9 nodeName:}" failed. No retries permitted until 2026-09-10 14:59:38.424845462 +0000 UTC m=+1048.901905387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-maas-api-tls") pod "maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276" (UID: "5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9") : secret "maas-api-e2e-shared-a-w3-16ad45-serving-cert" not found
Sep 10 14:59:37.925467 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.925444    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.927086 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.927067    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:37.933090 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:37.933068    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjhr\" (UniqueName: \"kubernetes.io/projected/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-kube-api-access-5vjhr\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:38.029060 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:38.028949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:38.126973 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:38.126940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:38.229474 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:38.229436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:38.326892 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:38.326803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:38.429031 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:38.429000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:38.431530 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:38.431481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:38.608901 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:38.608804    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:38.887239 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:38.887214    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"]
Sep 10 14:59:38.889498 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:59:38.889472    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5c48ed_d1d0_4a72_9995_5b1afe3f6ce9.slice/crio-08bade2f78aa299eb9cf8e5452a508516a8bc5d716c9d7c450d3159b54e36257 WatchSource:0}: Error finding container 08bade2f78aa299eb9cf8e5452a508516a8bc5d716c9d7c450d3159b54e36257: Status 404 returned error can't find the container with id 08bade2f78aa299eb9cf8e5452a508516a8bc5d716c9d7c450d3159b54e36257
Sep 10 14:59:39.140795 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:39.140764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276" event={"ID":"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9","Type":"ContainerStarted","Data":"08bade2f78aa299eb9cf8e5452a508516a8bc5d716c9d7c450d3159b54e36257"}
Sep 10 14:59:40.147398 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:40.147356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276" event={"ID":"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9","Type":"ContainerStarted","Data":"7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d"}
Sep 10 14:59:40.147948 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:40.147741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:40.166553 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:40.166488    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276" podStartSLOduration=2.848967847 podStartE2EDuration="3.166472879s" totalImagesPullingTime="317.505032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:40.164961504 +0000 UTC m=+1050.642021450" watchObservedRunningTime="2026-09-10 14:59:40.166472879 +0000 UTC m=+1050.643532823"
Sep 10 14:59:46.158189 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:46.158155    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 14:59:50.059667 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:50.059631    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:50.475070 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:50.475036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:51.049596 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:51.049565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:53.720104 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:53.720071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:53.980372 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:53.980277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:54.878314 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:54.878283    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"]
Sep 10 14:59:54.883432 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:54.883408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:54.885418 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:54.885386    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6621560b-serving-cert\""
Sep 10 14:59:54.895975 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:54.895948    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"]
Sep 10 14:59:54.912365 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:54.912328    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:54.982836 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:54.982806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-metrics-certs\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:54.982993 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:54.982885    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xphwk\" (UniqueName: \"kubernetes.io/projected/3aaa9244-634d-440d-87b5-c9f9ea3abd44-kube-api-access-xphwk\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:54.982993 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:54.982941    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9244-634d-440d-87b5-c9f9ea3abd44-openshift-service-ca\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:54.982993 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:54.982983    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-maas-api-tls\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.015305 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.015272    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:55.083967 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.083936    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-metrics-certs\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.084150 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.084007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xphwk\" (UniqueName: \"kubernetes.io/projected/3aaa9244-634d-440d-87b5-c9f9ea3abd44-kube-api-access-xphwk\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.084150 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.084029    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9244-634d-440d-87b5-c9f9ea3abd44-openshift-service-ca\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.084150 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.084052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-maas-api-tls\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.084316 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:59:55.084170    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-serving-cert: secret "maas-api-e2e-mt-6621560b-serving-cert" not found
Sep 10 14:59:55.084316 ip-10-0-141-186 kubenswrapper[2563]: E0910 14:59:55.084236    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-maas-api-tls podName:3aaa9244-634d-440d-87b5-c9f9ea3abd44 nodeName:}" failed. No retries permitted until 2026-09-10 14:59:55.584215975 +0000 UTC m=+1066.061275897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-maas-api-tls") pod "maas-api-e2e-mt-6621560b-68b56f7856-n5pd8" (UID: "3aaa9244-634d-440d-87b5-c9f9ea3abd44") : secret "maas-api-e2e-mt-6621560b-serving-cert" not found
Sep 10 14:59:55.084781 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.084760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9244-634d-440d-87b5-c9f9ea3abd44-openshift-service-ca\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.086528 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.086495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-metrics-certs\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.092329 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.092302    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xphwk\" (UniqueName: \"kubernetes.io/projected/3aaa9244-634d-440d-87b5-c9f9ea3abd44-kube-api-access-xphwk\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.430347 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.430315    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:55.528574 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.528537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 14:59:55.589209 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.589173    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-maas-api-tls\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.591574 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.591551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-maas-api-tls\") pod \"maas-api-e2e-mt-6621560b-68b56f7856-n5pd8\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.604060 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.604031    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"]
Sep 10 14:59:55.608039 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.608019    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 14:59:55.610172 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.610152    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6621560b\""
Sep 10 14:59:55.610331 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.610280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6621560b-dockercfg-f9fv7\""
Sep 10 14:59:55.620099 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.620077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"]
Sep 10 14:59:55.690532 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.690431    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0373f9c-10ce-4133-9b71-d22909704f8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62\" (UID: \"e0373f9c-10ce-4133-9b71-d22909704f8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 14:59:55.690794 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.690774    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rgd\" (UniqueName: \"kubernetes.io/projected/e0373f9c-10ce-4133-9b71-d22909704f8d-kube-api-access-54rgd\") pod \"payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62\" (UID: \"e0373f9c-10ce-4133-9b71-d22909704f8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 14:59:55.791834 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.791801    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0373f9c-10ce-4133-9b71-d22909704f8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62\" (UID: \"e0373f9c-10ce-4133-9b71-d22909704f8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 14:59:55.791834 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.791838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rgd\" (UniqueName: \"kubernetes.io/projected/e0373f9c-10ce-4133-9b71-d22909704f8d-kube-api-access-54rgd\") pod \"payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62\" (UID: \"e0373f9c-10ce-4133-9b71-d22909704f8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 14:59:55.792569 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.792550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0373f9c-10ce-4133-9b71-d22909704f8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62\" (UID: \"e0373f9c-10ce-4133-9b71-d22909704f8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 14:59:55.795655 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.795632    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:55.799966 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.799947    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rgd\" (UniqueName: \"kubernetes.io/projected/e0373f9c-10ce-4133-9b71-d22909704f8d-kube-api-access-54rgd\") pod \"payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62\" (UID: \"e0373f9c-10ce-4133-9b71-d22909704f8d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 14:59:55.920200 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:55.920170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 14:59:56.074620 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:56.074594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"]
Sep 10 14:59:56.077212 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:59:56.077189    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aaa9244_634d_440d_87b5_c9f9ea3abd44.slice/crio-7f50e12e4d46dd337047c702e2004acec74bae442e042501ebf2677c8b35d8ac WatchSource:0}: Error finding container 7f50e12e4d46dd337047c702e2004acec74bae442e042501ebf2677c8b35d8ac: Status 404 returned error can't find the container with id 7f50e12e4d46dd337047c702e2004acec74bae442e042501ebf2677c8b35d8ac
Sep 10 14:59:56.198656 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:56.198634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"]
Sep 10 14:59:56.200415 ip-10-0-141-186 kubenswrapper[2563]: W0910 14:59:56.200394    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0373f9c_10ce_4133_9b71_d22909704f8d.slice/crio-d7e3ca249928013118c4e5705ab26eb56540ea35c67bb5fff32e398a45a9f3c3 WatchSource:0}: Error finding container d7e3ca249928013118c4e5705ab26eb56540ea35c67bb5fff32e398a45a9f3c3: Status 404 returned error can't find the container with id d7e3ca249928013118c4e5705ab26eb56540ea35c67bb5fff32e398a45a9f3c3
Sep 10 14:59:56.238176 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:56.238075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62" event={"ID":"e0373f9c-10ce-4133-9b71-d22909704f8d","Type":"ContainerStarted","Data":"d7e3ca249928013118c4e5705ab26eb56540ea35c67bb5fff32e398a45a9f3c3"}
Sep 10 14:59:56.239405 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:56.239378    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8" event={"ID":"3aaa9244-634d-440d-87b5-c9f9ea3abd44","Type":"ContainerStarted","Data":"7f50e12e4d46dd337047c702e2004acec74bae442e042501ebf2677c8b35d8ac"}
Sep 10 14:59:57.246040 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:57.246006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62" event={"ID":"e0373f9c-10ce-4133-9b71-d22909704f8d","Type":"ContainerStarted","Data":"02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293"}
Sep 10 14:59:57.246546 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:57.246125    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 14:59:57.247659 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:57.247628    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8" event={"ID":"3aaa9244-634d-440d-87b5-c9f9ea3abd44","Type":"ContainerStarted","Data":"aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5"}
Sep 10 14:59:57.247786 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:57.247690    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 14:59:57.282336 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:57.282290    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62" podStartSLOduration=2.282275252 podStartE2EDuration="2.282275252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:57.263097575 +0000 UTC m=+1067.740157556" watchObservedRunningTime="2026-09-10 14:59:57.282275252 +0000 UTC m=+1067.759335196"
Sep 10 14:59:57.283003 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:57.282977    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8" podStartSLOduration=2.916985617 podStartE2EDuration="3.28296718s" totalImagesPullingTime="365.981563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:59:57.281202137 +0000 UTC m=+1067.758262091" watchObservedRunningTime="2026-09-10 14:59:57.28296718 +0000 UTC m=+1067.760027123"
Sep 10 14:59:58.640477 ip-10-0-141-186 kubenswrapper[2563]: I0910 14:59:58.640433    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:01.623311 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:01.623264    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:02.034002 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:02.033970    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:02.322860 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:02.322775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:03.129676 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:03.129642    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:03.260568 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:03.260544    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 15:00:05.975164 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:05.975117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:06.084787 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:06.084754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:07.034578 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.034541    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"]
Sep 10 15:00:07.039476 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.039454    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.041903 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.041880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-3f81b7-serving-cert\""
Sep 10 15:00:07.052666 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.052643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"]
Sep 10 15:00:07.093024 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.092995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:07.103238 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.103212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzfx\" (UniqueName: \"kubernetes.io/projected/b78d2fb4-8068-48fc-87b7-32b6a0086f79-kube-api-access-mdzfx\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.103395 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.103270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.103395 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.103326    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.103550 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.103452    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78d2fb4-8068-48fc-87b7-32b6a0086f79-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.192824 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.192792    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:07.204927 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.204898    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.205065 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.204965    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78d2fb4-8068-48fc-87b7-32b6a0086f79-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.205065 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.205019    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzfx\" (UniqueName: \"kubernetes.io/projected/b78d2fb4-8068-48fc-87b7-32b6a0086f79-kube-api-access-mdzfx\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.205065 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:00:07.205057    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-serving-cert: secret "maas-api-e2e-shared-b-w3-3f81b7-serving-cert" not found
Sep 10 15:00:07.205173 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.205067    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.205173 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:00:07.205130    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-maas-api-tls podName:b78d2fb4-8068-48fc-87b7-32b6a0086f79 nodeName:}" failed. No retries permitted until 2026-09-10 15:00:07.705110292 +0000 UTC m=+1078.182170227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-maas-api-tls") pod "maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7" (UID: "b78d2fb4-8068-48fc-87b7-32b6a0086f79") : secret "maas-api-e2e-shared-b-w3-3f81b7-serving-cert" not found
Sep 10 15:00:07.205731 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.205708    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78d2fb4-8068-48fc-87b7-32b6a0086f79-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.207479 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.207459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.216655 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.216631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzfx\" (UniqueName: \"kubernetes.io/projected/b78d2fb4-8068-48fc-87b7-32b6a0086f79-kube-api-access-mdzfx\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.619922 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.619887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:07.710333 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.710291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.713182 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.713140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:07.954419 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:07.954383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:08.142710 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:08.142649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:08.234881 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:08.234712    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"]
Sep 10 15:00:08.237230 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:00:08.237201    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78d2fb4_8068_48fc_87b7_32b6a0086f79.slice/crio-85d25409a7cfe181da3d209d0cd4bd9a78d670a8372e83a1acdfd59b9cff9e6d WatchSource:0}: Error finding container 85d25409a7cfe181da3d209d0cd4bd9a78d670a8372e83a1acdfd59b9cff9e6d: Status 404 returned error can't find the container with id 85d25409a7cfe181da3d209d0cd4bd9a78d670a8372e83a1acdfd59b9cff9e6d
Sep 10 15:00:08.238773 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:08.238755    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:00:08.240424 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:08.240400    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:08.256922 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:08.256901    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 15:00:08.308637 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:08.308554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7" event={"ID":"b78d2fb4-8068-48fc-87b7-32b6a0086f79","Type":"ContainerStarted","Data":"85d25409a7cfe181da3d209d0cd4bd9a78d670a8372e83a1acdfd59b9cff9e6d"}
Sep 10 15:00:09.316162 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:09.316123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7" event={"ID":"b78d2fb4-8068-48fc-87b7-32b6a0086f79","Type":"ContainerStarted","Data":"862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06"}
Sep 10 15:00:09.316656 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:09.316261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:09.334630 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:09.334581    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7" podStartSLOduration=1.967847888 podStartE2EDuration="2.334566365s" totalImagesPullingTime="366.718477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:00:09.333752193 +0000 UTC m=+1079.810812138" watchObservedRunningTime="2026-09-10 15:00:09.334566365 +0000 UTC m=+1079.811626309"
Sep 10 15:00:15.327522 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:15.327479    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:00:16.024199 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:16.024160    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:18.683178 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:18.683147    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:22.708841 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:22.708772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:28.434450 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:28.434408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:30.894988 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:30.894953    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:39.255159 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:39.255124    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:00:50.199187 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:00:50.199149    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:01:08.742362 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:08.742323    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:01:13.661310 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:13.661277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:01:17.694527 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:17.694478    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:01:24.203300 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.203263    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"]
Sep 10 15:01:24.204062 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.203538    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8" podUID="3aaa9244-634d-440d-87b5-c9f9ea3abd44" containerName="maas-api" containerID="cri-o://aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5" gracePeriod=30
Sep 10 15:01:24.236847 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.236822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:01:24.271406 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.271379    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"]
Sep 10 15:01:24.271635 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.271608    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62" podUID="e0373f9c-10ce-4133-9b71-d22909704f8d" containerName="payload-pre-processing" containerID="cri-o://02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293" gracePeriod=30
Sep 10 15:01:24.337235 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.337204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:01:24.439697 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.439657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:01:24.505097 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.505065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:01:24.541255 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.541230    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 15:01:24.547889 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.547864    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-metrics-certs\") pod \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") "
Sep 10 15:01:24.548040 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.547961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xphwk\" (UniqueName: \"kubernetes.io/projected/3aaa9244-634d-440d-87b5-c9f9ea3abd44-kube-api-access-xphwk\") pod \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") "
Sep 10 15:01:24.548104 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.548045    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-maas-api-tls\") pod \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") "
Sep 10 15:01:24.548104 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.548081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9244-634d-440d-87b5-c9f9ea3abd44-openshift-service-ca\") pod \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\" (UID: \"3aaa9244-634d-440d-87b5-c9f9ea3abd44\") "
Sep 10 15:01:24.548689 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.548656    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aaa9244-634d-440d-87b5-c9f9ea3abd44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3aaa9244-634d-440d-87b5-c9f9ea3abd44" (UID: "3aaa9244-634d-440d-87b5-c9f9ea3abd44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:01:24.550333 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.550278    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3aaa9244-634d-440d-87b5-c9f9ea3abd44" (UID: "3aaa9244-634d-440d-87b5-c9f9ea3abd44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:01:24.550333 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.550237    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aaa9244-634d-440d-87b5-c9f9ea3abd44-kube-api-access-xphwk" (OuterVolumeSpecName: "kube-api-access-xphwk") pod "3aaa9244-634d-440d-87b5-c9f9ea3abd44" (UID: "3aaa9244-634d-440d-87b5-c9f9ea3abd44"). InnerVolumeSpecName "kube-api-access-xphwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:01:24.550495 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.550361    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3aaa9244-634d-440d-87b5-c9f9ea3abd44" (UID: "3aaa9244-634d-440d-87b5-c9f9ea3abd44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:01:24.574606 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.574581    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 15:01:24.649640 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.649608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54rgd\" (UniqueName: \"kubernetes.io/projected/e0373f9c-10ce-4133-9b71-d22909704f8d-kube-api-access-54rgd\") pod \"e0373f9c-10ce-4133-9b71-d22909704f8d\" (UID: \"e0373f9c-10ce-4133-9b71-d22909704f8d\") "
Sep 10 15:01:24.649824 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.649657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0373f9c-10ce-4133-9b71-d22909704f8d-plugins-config-volume\") pod \"e0373f9c-10ce-4133-9b71-d22909704f8d\" (UID: \"e0373f9c-10ce-4133-9b71-d22909704f8d\") "
Sep 10 15:01:24.649901 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.649860    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:01:24.649901 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.649871    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aaa9244-634d-440d-87b5-c9f9ea3abd44-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:01:24.649901 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.649881    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aaa9244-634d-440d-87b5-c9f9ea3abd44-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:01:24.649901 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.649890    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xphwk\" (UniqueName: \"kubernetes.io/projected/3aaa9244-634d-440d-87b5-c9f9ea3abd44-kube-api-access-xphwk\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:01:24.650098 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.650071    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0373f9c-10ce-4133-9b71-d22909704f8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0373f9c-10ce-4133-9b71-d22909704f8d" (UID: "e0373f9c-10ce-4133-9b71-d22909704f8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:01:24.651757 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.651732    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0373f9c-10ce-4133-9b71-d22909704f8d-kube-api-access-54rgd" (OuterVolumeSpecName: "kube-api-access-54rgd") pod "e0373f9c-10ce-4133-9b71-d22909704f8d" (UID: "e0373f9c-10ce-4133-9b71-d22909704f8d"). InnerVolumeSpecName "kube-api-access-54rgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:01:24.716477 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.716439    2563 generic.go:356] "Generic (PLEG): container finished" podID="e0373f9c-10ce-4133-9b71-d22909704f8d" containerID="02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293" exitCode=0
Sep 10 15:01:24.716689 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.716494    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"
Sep 10 15:01:24.716689 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.716559    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62" event={"ID":"e0373f9c-10ce-4133-9b71-d22909704f8d","Type":"ContainerDied","Data":"02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293"}
Sep 10 15:01:24.716689 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.716646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62" event={"ID":"e0373f9c-10ce-4133-9b71-d22909704f8d","Type":"ContainerDied","Data":"d7e3ca249928013118c4e5705ab26eb56540ea35c67bb5fff32e398a45a9f3c3"}
Sep 10 15:01:24.716689 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.716683    2563 scope.go:117] "RemoveContainer" containerID="02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293"
Sep 10 15:01:24.718222 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.718198    2563 generic.go:356] "Generic (PLEG): container finished" podID="3aaa9244-634d-440d-87b5-c9f9ea3abd44" containerID="aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5" exitCode=0
Sep 10 15:01:24.718358 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.718237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8" event={"ID":"3aaa9244-634d-440d-87b5-c9f9ea3abd44","Type":"ContainerDied","Data":"aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5"}
Sep 10 15:01:24.718358 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.718247    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"
Sep 10 15:01:24.718358 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.718279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8" event={"ID":"3aaa9244-634d-440d-87b5-c9f9ea3abd44","Type":"ContainerDied","Data":"7f50e12e4d46dd337047c702e2004acec74bae442e042501ebf2677c8b35d8ac"}
Sep 10 15:01:24.726232 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.726214    2563 scope.go:117] "RemoveContainer" containerID="02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293"
Sep 10 15:01:24.726569 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:01:24.726543    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293\": container with ID starting with 02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293 not found: ID does not exist" containerID="02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293"
Sep 10 15:01:24.726650 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.726580    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293"} err="failed to get container status \"02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293\": rpc error: code = NotFound desc = could not find container \"02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293\": container with ID starting with 02dda26605ef2d14b3b874f905c8b2c129f9466c774a7bb56acc7f79bb3ba293 not found: ID does not exist"
Sep 10 15:01:24.726650 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.726602    2563 scope.go:117] "RemoveContainer" containerID="aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5"
Sep 10 15:01:24.735028 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.735009    2563 scope.go:117] "RemoveContainer" containerID="aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5"
Sep 10 15:01:24.735300 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:01:24.735283    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5\": container with ID starting with aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5 not found: ID does not exist" containerID="aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5"
Sep 10 15:01:24.735363 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.735306    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5"} err="failed to get container status \"aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5\": rpc error: code = NotFound desc = could not find container \"aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5\": container with ID starting with aded915dbf11ff50c48b98d78e9b3b46e76749bbf1ebf3ee3074bd1cb68dd8f5 not found: ID does not exist"
Sep 10 15:01:24.743050 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.742997    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"]
Sep 10 15:01:24.750194 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.750174    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6621560b-7c79d7ddf4-xfz62"]
Sep 10 15:01:24.750540 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.750523    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54rgd\" (UniqueName: \"kubernetes.io/projected/e0373f9c-10ce-4133-9b71-d22909704f8d-kube-api-access-54rgd\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:01:24.750608 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.750544    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0373f9c-10ce-4133-9b71-d22909704f8d-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:01:24.764556 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.764536    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"]
Sep 10 15:01:24.768571 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:24.768554    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6621560b-68b56f7856-n5pd8"]
Sep 10 15:01:26.033255 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:26.033224    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aaa9244-634d-440d-87b5-c9f9ea3abd44" path="/var/lib/kubelet/pods/3aaa9244-634d-440d-87b5-c9f9ea3abd44/volumes"
Sep 10 15:01:26.033637 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:26.033600    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0373f9c-10ce-4133-9b71-d22909704f8d" path="/var/lib/kubelet/pods/e0373f9c-10ce-4133-9b71-d22909704f8d/volumes"
Sep 10 15:01:27.820272 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:27.820239    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:01:32.987348 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:01:32.987246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:09.219279 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:09.219248    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:10.038924 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:10.038893    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 15:02:10.047815 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:10.047788    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 15:02:10.191993 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:10.191969    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:16.204672 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:16.204636    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:17.814427 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:17.814391    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:17.989964 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:17.989924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:18.615797 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:18.615760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:18.715610 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:18.715573    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:20.117556 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:20.117498    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:26.222581 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:26.222546    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:26.353616 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:26.353586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:26.932799 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:26.932768    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:34.190672 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:34.190621    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:34.430797 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:34.430766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:34.626463 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:34.626359    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:44.981575 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:44.981537    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:02:45.225242 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:02:45.225204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:06.168620 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:06.168533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:11.053728 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:11.053698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:11.887860 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:11.887821    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:12.744632 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:12.744596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:12.847782 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:12.847751    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:13.088199 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.088116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:13.368969 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.368889    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"]
Sep 10 15:03:13.369377 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.369362    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aaa9244-634d-440d-87b5-c9f9ea3abd44" containerName="maas-api"
Sep 10 15:03:13.369434 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.369381    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aaa9244-634d-440d-87b5-c9f9ea3abd44" containerName="maas-api"
Sep 10 15:03:13.369434 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.369396    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0373f9c-10ce-4133-9b71-d22909704f8d" containerName="payload-pre-processing"
Sep 10 15:03:13.369434 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.369405    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0373f9c-10ce-4133-9b71-d22909704f8d" containerName="payload-pre-processing"
Sep 10 15:03:13.369576 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.369518    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aaa9244-634d-440d-87b5-c9f9ea3abd44" containerName="maas-api"
Sep 10 15:03:13.369576 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.369532    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0373f9c-10ce-4133-9b71-d22909704f8d" containerName="payload-pre-processing"
Sep 10 15:03:13.373209 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.373183    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.375565 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.375543    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-550dc3-openshift-default-dockercfg-cz4v6\""
Sep 10 15:03:13.385550 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.385526    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"]
Sep 10 15:03:13.400011 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.399986    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:13.483244 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.483202    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-certs\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.483434 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.483257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-credential-socket\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.483434 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.483280    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-token\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.483434 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.483311    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-data\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.483434 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.483333    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-podinfo\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.483434 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.483398    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istiod-ca-cert\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.483683 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.483440    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-envoy\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.483683 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.483473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-socket\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.483683 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.483490    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmn2f\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-kube-api-access-zmn2f\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.582857 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.582816    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:13.584815 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.584793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-data\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.584875 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.584825    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-podinfo\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.584875 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.584860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istiod-ca-cert\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.584947 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.584883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-envoy\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.584947 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.584920    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-socket\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.585016 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.584943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmn2f\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-kube-api-access-zmn2f\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.585064 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.585013    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-certs\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.585064 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.585056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-credential-socket\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.585145 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.585083    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-token\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.585248 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.585227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-data\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.585303 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.585288    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-socket\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.585555 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.585495    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-certs\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.585673 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.585615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-credential-socket\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.585717 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.585672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istiod-ca-cert\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.587349 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.587330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-envoy\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.587687 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.587669    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-podinfo\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.592881 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.592863    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmn2f\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-kube-api-access-zmn2f\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.593059 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.593038    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-token\") pod \"e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") " pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.684867 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.684835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:13.968874 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.968850    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"]
Sep 10 15:03:13.971092 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:03:13.971063    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cb9000_2ebf_4164_8d3e_1cfae086420b.slice/crio-0f37abaf22da7a835bebaff708f046e6fdb4039ba2b6027664665931b7963cdf WatchSource:0}: Error finding container 0f37abaf22da7a835bebaff708f046e6fdb4039ba2b6027664665931b7963cdf: Status 404 returned error can't find the container with id 0f37abaf22da7a835bebaff708f046e6fdb4039ba2b6027664665931b7963cdf
Sep 10 15:03:13.973468 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.973428    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:03:13.973568 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.973534    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:03:13.973618 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:13.973575    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:03:14.293854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:14.293763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" event={"ID":"f8cb9000-2ebf-4164-8d3e-1cfae086420b","Type":"ContainerStarted","Data":"cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f"}
Sep 10 15:03:14.293854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:14.293822    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" event={"ID":"f8cb9000-2ebf-4164-8d3e-1cfae086420b","Type":"ContainerStarted","Data":"0f37abaf22da7a835bebaff708f046e6fdb4039ba2b6027664665931b7963cdf"}
Sep 10 15:03:14.314416 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:14.314369    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" podStartSLOduration=1.3143523990000001 podStartE2EDuration="1.314352399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:03:14.311989856 +0000 UTC m=+1264.789049799" watchObservedRunningTime="2026-09-10 15:03:14.314352399 +0000 UTC m=+1264.791412418"
Sep 10 15:03:14.685478 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:14.685440    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:15.690275 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:15.690246    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:16.271973 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:16.271925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:16.304563 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:16.304500    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:16.305435 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:16.305417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:16.361603 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:16.361570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:17.613952 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:17.613909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:17.678225 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:17.678195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:17.881940 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:17.881850    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:18.134019 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:18.133930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:18.229883 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:18.229848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:31.041744 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:31.041705    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:31.198034 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:31.197995    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:31.299714 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:31.299622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:32.345667 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:32.345631    2563 patch_prober.go:28] interesting pod/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 15:03:32.346075 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:32.345717    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 15:03:33.323380 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:33.323277    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:33.401726 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:33.401695    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerID="cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f" exitCode=0
Sep 10 15:03:33.402202 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:33.401767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" event={"ID":"f8cb9000-2ebf-4164-8d3e-1cfae086420b","Type":"ContainerDied","Data":"cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f"}
Sep 10 15:03:33.402288 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:33.402269    2563 scope.go:117] "RemoveContainer" containerID="cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f"
Sep 10 15:03:33.403464 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:33.403439    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:03:33.403552 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:33.403540    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:03:33.403598 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:33.403573    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:03:33.880873 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:33.880837    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:34.409474 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:34.409434    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" event={"ID":"f8cb9000-2ebf-4164-8d3e-1cfae086420b","Type":"ContainerStarted","Data":"d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f"}
Sep 10 15:03:34.685979 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:34.685902    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:34.686229 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:34.686208    2563 patch_prober.go:28] interesting pod/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 15:03:34.686324 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:34.686261    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 15:03:35.685536 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:35.685479    2563 patch_prober.go:28] interesting pod/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 15:03:35.685950 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:35.685570    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 15:03:36.685653 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:36.685618    2563 patch_prober.go:28] interesting pod/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 15:03:36.686059 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:36.685697    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 15:03:37.686032 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:37.685988    2563 patch_prober.go:28] interesting pod/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 15:03:37.686496 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:37.686063    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 15:03:38.685441 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:38.685407    2563 patch_prober.go:28] interesting pod/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 15:03:38.685653 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:38.685459    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 15:03:39.686151 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:39.686114    2563 patch_prober.go:28] interesting pod/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 10 15:03:39.686588 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:39.686179    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 10 15:03:40.690412 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:40.690382    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:40.690842 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:40.690723    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:40.691469 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:40.691452    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:03:41.270752 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:41.270719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:41.718834 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:41.718798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:42.041106 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:42.041020    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:42.544100 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:42.544035    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:45.235987 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:45.235955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:45.333583 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:45.333543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:46.478248 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.478215    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"]
Sep 10 15:03:46.482700 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.482678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.485136 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.485111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cb65dc-serving-cert\""
Sep 10 15:03:46.494951 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.494925    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"]
Sep 10 15:03:46.516199 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.516170    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlcm\" (UniqueName: \"kubernetes.io/projected/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-kube-api-access-8zlcm\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.516382 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.516209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-openshift-service-ca\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.516382 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.516229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-metrics-certs\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.516529 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.516383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-maas-api-tls\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.526560 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.526523    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:46.617795 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.617764    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-maas-api-tls\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.618002 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.617846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlcm\" (UniqueName: \"kubernetes.io/projected/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-kube-api-access-8zlcm\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.618002 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.617897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-openshift-service-ca\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.618002 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.617925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-metrics-certs\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.618002 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:03:46.617951    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-serving-cert: secret "maas-api-e2e-api-b-cb65dc-serving-cert" not found
Sep 10 15:03:46.618214 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:03:46.618023    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-maas-api-tls podName:b829c2ab-4cbf-4ab9-9148-7f52e369b32a nodeName:}" failed. No retries permitted until 2026-09-10 15:03:47.118003191 +0000 UTC m=+1297.595063113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-maas-api-tls") pod "maas-api-e2e-api-b-cb65dc-66659598fc-q95zh" (UID: "b829c2ab-4cbf-4ab9-9148-7f52e369b32a") : secret "maas-api-e2e-api-b-cb65dc-serving-cert" not found
Sep 10 15:03:46.618610 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.618581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-openshift-service-ca\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.620611 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.620589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-metrics-certs\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.627022 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.626993    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:46.627315 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.627297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlcm\" (UniqueName: \"kubernetes.io/projected/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-kube-api-access-8zlcm\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:46.784118 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:46.784034    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:47.043948 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.043858    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:47.124090 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.124054    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-maas-api-tls\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:47.126778 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.126748    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-maas-api-tls\") pod \"maas-api-e2e-api-b-cb65dc-66659598fc-q95zh\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:47.133723 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.133697    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:03:47.180956 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.180922    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"]
Sep 10 15:03:47.184655 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.184633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:03:47.187432 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.187403    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cb65dc-dockercfg-nb4kp\""
Sep 10 15:03:47.187432 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.187421    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cb65dc\""
Sep 10 15:03:47.197135 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.197113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"]
Sep 10 15:03:47.224794 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.224763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7d9d\" (UniqueName: \"kubernetes.io/projected/522c68ff-5517-41f0-b9a0-2759155ac29f-kube-api-access-x7d9d\") pod \"payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp\" (UID: \"522c68ff-5517-41f0-b9a0-2759155ac29f\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:03:47.224975 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.224898    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c68ff-5517-41f0-b9a0-2759155ac29f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp\" (UID: \"522c68ff-5517-41f0-b9a0-2759155ac29f\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:03:47.326196 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.326119    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7d9d\" (UniqueName: \"kubernetes.io/projected/522c68ff-5517-41f0-b9a0-2759155ac29f-kube-api-access-x7d9d\") pod \"payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp\" (UID: \"522c68ff-5517-41f0-b9a0-2759155ac29f\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:03:47.326340 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.326199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c68ff-5517-41f0-b9a0-2759155ac29f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp\" (UID: \"522c68ff-5517-41f0-b9a0-2759155ac29f\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:03:47.326937 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.326919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c68ff-5517-41f0-b9a0-2759155ac29f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp\" (UID: \"522c68ff-5517-41f0-b9a0-2759155ac29f\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:03:47.334502 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.334474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7d9d\" (UniqueName: \"kubernetes.io/projected/522c68ff-5517-41f0-b9a0-2759155ac29f-kube-api-access-x7d9d\") pod \"payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp\" (UID: \"522c68ff-5517-41f0-b9a0-2759155ac29f\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:03:47.395803 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.395776    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:47.496696 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.496660    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:03:47.684798 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.684774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"]
Sep 10 15:03:47.686777 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:03:47.686736    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb829c2ab_4cbf_4ab9_9148_7f52e369b32a.slice/crio-267389aafcc69c0dd9ded53ce7f8657dfcb7d0599a8a68579188df877098010d WatchSource:0}: Error finding container 267389aafcc69c0dd9ded53ce7f8657dfcb7d0599a8a68579188df877098010d: Status 404 returned error can't find the container with id 267389aafcc69c0dd9ded53ce7f8657dfcb7d0599a8a68579188df877098010d
Sep 10 15:03:47.772543 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:47.772520    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"]
Sep 10 15:03:47.774577 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:03:47.774551    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522c68ff_5517_41f0_b9a0_2759155ac29f.slice/crio-e2c43ff65948b13712d5808e4d33ceb0bbb01e44500b1e6762c9e4bea507e797 WatchSource:0}: Error finding container e2c43ff65948b13712d5808e4d33ceb0bbb01e44500b1e6762c9e4bea507e797: Status 404 returned error can't find the container with id e2c43ff65948b13712d5808e4d33ceb0bbb01e44500b1e6762c9e4bea507e797
Sep 10 15:03:48.497490 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:48.497456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh" event={"ID":"b829c2ab-4cbf-4ab9-9148-7f52e369b32a","Type":"ContainerStarted","Data":"2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1"}
Sep 10 15:03:48.497959 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:48.497535    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:48.497959 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:48.497566    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh" event={"ID":"b829c2ab-4cbf-4ab9-9148-7f52e369b32a","Type":"ContainerStarted","Data":"267389aafcc69c0dd9ded53ce7f8657dfcb7d0599a8a68579188df877098010d"}
Sep 10 15:03:48.499050 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:48.499027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp" event={"ID":"522c68ff-5517-41f0-b9a0-2759155ac29f","Type":"ContainerStarted","Data":"0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b"}
Sep 10 15:03:48.499192 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:48.499066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp" event={"ID":"522c68ff-5517-41f0-b9a0-2759155ac29f","Type":"ContainerStarted","Data":"e2c43ff65948b13712d5808e4d33ceb0bbb01e44500b1e6762c9e4bea507e797"}
Sep 10 15:03:48.499192 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:48.499101    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:03:48.518990 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:48.518946    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh" podStartSLOduration=2.149051005 podStartE2EDuration="2.518927067s" totalImagesPullingTime="369.876062ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:03:48.517815924 +0000 UTC m=+1298.994875882" watchObservedRunningTime="2026-09-10 15:03:48.518927067 +0000 UTC m=+1298.995987015"
Sep 10 15:03:48.537636 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:48.537589    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp" podStartSLOduration=1.5375728579999999 podStartE2EDuration="1.537572858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:03:48.536239553 +0000 UTC m=+1299.013299497" watchObservedRunningTime="2026-09-10 15:03:48.537572858 +0000 UTC m=+1299.014632801"
Sep 10 15:03:54.511746 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:54.511716    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:03:59.507230 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:03:59.507199    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:04:02.257957 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:02.257924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:08.127604 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:08.127565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:08.865182 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:08.865143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:10.317825 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:10.317793    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:13.328442 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:13.328405    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:28.770164 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:28.770128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:28.838026 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:28.837994    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:28.905308 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:28.905263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:28.992308 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:28.992278    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:29.065327 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:29.065238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:29.375883 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:29.375798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:35.177458 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:35.177369    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:35.240744 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:35.240713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:35.418659 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:35.418625    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"]
Sep 10 15:04:35.418989 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:35.418918    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy" containerID="cri-o://d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f" gracePeriod=30
Sep 10 15:04:35.440898 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:35.440817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:35.913667 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:35.913629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:36.761488 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:36.761449    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:37.506786 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:37.506754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:40.665917 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.665892    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:04:40.729077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.728989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-token\") pod \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") "
Sep 10 15:04:40.729077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729047    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmn2f\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-kube-api-access-zmn2f\") pod \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") "
Sep 10 15:04:40.729276 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729216    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-data\") pod \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") "
Sep 10 15:04:40.729318 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-envoy\") pod \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") "
Sep 10 15:04:40.729370 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729318    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-socket\") pod \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") "
Sep 10 15:04:40.729370 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729347    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-credential-socket\") pod \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") "
Sep 10 15:04:40.729480 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729373    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-certs\") pod \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") "
Sep 10 15:04:40.729480 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istiod-ca-cert\") pod \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") "
Sep 10 15:04:40.729480 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729467    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8cb9000-2ebf-4164-8d3e-1cfae086420b" (UID: "f8cb9000-2ebf-4164-8d3e-1cfae086420b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:40.729698 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-podinfo\") pod \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\" (UID: \"f8cb9000-2ebf-4164-8d3e-1cfae086420b\") "
Sep 10 15:04:40.729767 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729734    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8cb9000-2ebf-4164-8d3e-1cfae086420b" (UID: "f8cb9000-2ebf-4164-8d3e-1cfae086420b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:40.729822 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.729775    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8cb9000-2ebf-4164-8d3e-1cfae086420b" (UID: "f8cb9000-2ebf-4164-8d3e-1cfae086420b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:40.730027 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.730004    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8cb9000-2ebf-4164-8d3e-1cfae086420b" (UID: "f8cb9000-2ebf-4164-8d3e-1cfae086420b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:04:40.730121 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.730097    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8cb9000-2ebf-4164-8d3e-1cfae086420b" (UID: "f8cb9000-2ebf-4164-8d3e-1cfae086420b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:40.730271 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.730251    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istiod-ca-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:40.730337 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.730277    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-data\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:40.730337 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.730292    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:40.730337 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.730306    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-credential-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:40.730337 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.730321    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-workload-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:40.731650 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.731616    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8cb9000-2ebf-4164-8d3e-1cfae086420b" (UID: "f8cb9000-2ebf-4164-8d3e-1cfae086420b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:04:40.731650 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.731624    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-kube-api-access-zmn2f" (OuterVolumeSpecName: "kube-api-access-zmn2f") pod "f8cb9000-2ebf-4164-8d3e-1cfae086420b" (UID: "f8cb9000-2ebf-4164-8d3e-1cfae086420b"). InnerVolumeSpecName "kube-api-access-zmn2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:40.731826 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.731710    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8cb9000-2ebf-4164-8d3e-1cfae086420b" (UID: "f8cb9000-2ebf-4164-8d3e-1cfae086420b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:04:40.731826 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.731713    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8cb9000-2ebf-4164-8d3e-1cfae086420b" (UID: "f8cb9000-2ebf-4164-8d3e-1cfae086420b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:40.785595 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.785548    2563 generic.go:356] "Generic (PLEG): container finished" podID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerID="d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f" exitCode=0
Sep 10 15:04:40.785595 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.785596    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"
Sep 10 15:04:40.785792 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.785630    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" event={"ID":"f8cb9000-2ebf-4164-8d3e-1cfae086420b","Type":"ContainerDied","Data":"d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f"}
Sep 10 15:04:40.785792 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.785691    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7" event={"ID":"f8cb9000-2ebf-4164-8d3e-1cfae086420b","Type":"ContainerDied","Data":"0f37abaf22da7a835bebaff708f046e6fdb4039ba2b6027664665931b7963cdf"}
Sep 10 15:04:40.785792 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.785716    2563 scope.go:117] "RemoveContainer" containerID="d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f"
Sep 10 15:04:40.797336 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.797244    2563 scope.go:117] "RemoveContainer" containerID="cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f"
Sep 10 15:04:40.806895 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.806877    2563 scope.go:117] "RemoveContainer" containerID="d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f"
Sep 10 15:04:40.807151 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.807130    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"]
Sep 10 15:04:40.807227 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:04:40.807156    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f\": container with ID starting with d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f not found: ID does not exist" containerID="d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f"
Sep 10 15:04:40.807227 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.807192    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f"} err="failed to get container status \"d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f\": rpc error: code = NotFound desc = could not find container \"d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f\": container with ID starting with d99b32fe23cde27e4c3e5f43f8f439af202763771a0e0e3fc21f21637916aa9f not found: ID does not exist"
Sep 10 15:04:40.807227 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.807216    2563 scope.go:117] "RemoveContainer" containerID="cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f"
Sep 10 15:04:40.807460 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:04:40.807444    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f\": container with ID starting with cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f not found: ID does not exist" containerID="cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f"
Sep 10 15:04:40.807531 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.807471    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f"} err="failed to get container status \"cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f\": rpc error: code = NotFound desc = could not find container \"cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f\": container with ID starting with cc436ac0205b619964df23ac30d2f7e650c6b792c052a78bbc6bfbdf5e587f6f not found: ID does not exist"
Sep 10 15:04:40.811204 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.811181    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-550dc3-openshift-default-85d57474d-rgdz7"]
Sep 10 15:04:40.830829 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.830806    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-token\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:40.830829 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.830829    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmn2f\" (UniqueName: \"kubernetes.io/projected/f8cb9000-2ebf-4164-8d3e-1cfae086420b-kube-api-access-zmn2f\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:40.830992 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.830839    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-envoy\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:40.830992 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:40.830850    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8cb9000-2ebf-4164-8d3e-1cfae086420b-istio-podinfo\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:42.033403 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:42.033358    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" path="/var/lib/kubelet/pods/f8cb9000-2ebf-4164-8d3e-1cfae086420b/volumes"
Sep 10 15:04:47.671772 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:47.671735    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:47.737136 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:47.737105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:51.386274 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.386239    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"]
Sep 10 15:04:51.386703 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.386473    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh" podUID="b829c2ab-4cbf-4ab9-9148-7f52e369b32a" containerName="maas-api" containerID="cri-o://2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1" gracePeriod=30
Sep 10 15:04:51.415758 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.415695    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:51.444894 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.444859    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"]
Sep 10 15:04:51.445140 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.445112    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp" podUID="522c68ff-5517-41f0-b9a0-2759155ac29f" containerName="payload-processing" containerID="cri-o://0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b" gracePeriod=30
Sep 10 15:04:51.517273 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.517232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:51.617255 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.616940    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:51.699296 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.699266    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:04:51.723717 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.723481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:51.732016 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.731992    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:04:51.840060 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840032    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-openshift-service-ca\") pod \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") "
Sep 10 15:04:51.840230 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840080    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c68ff-5517-41f0-b9a0-2759155ac29f-plugins-config-volume\") pod \"522c68ff-5517-41f0-b9a0-2759155ac29f\" (UID: \"522c68ff-5517-41f0-b9a0-2759155ac29f\") "
Sep 10 15:04:51.840230 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zlcm\" (UniqueName: \"kubernetes.io/projected/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-kube-api-access-8zlcm\") pod \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") "
Sep 10 15:04:51.840230 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7d9d\" (UniqueName: \"kubernetes.io/projected/522c68ff-5517-41f0-b9a0-2759155ac29f-kube-api-access-x7d9d\") pod \"522c68ff-5517-41f0-b9a0-2759155ac29f\" (UID: \"522c68ff-5517-41f0-b9a0-2759155ac29f\") "
Sep 10 15:04:51.840230 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-maas-api-tls\") pod \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") "
Sep 10 15:04:51.840465 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840251    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-metrics-certs\") pod \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\" (UID: \"b829c2ab-4cbf-4ab9-9148-7f52e369b32a\") "
Sep 10 15:04:51.840563 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840488    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b829c2ab-4cbf-4ab9-9148-7f52e369b32a" (UID: "b829c2ab-4cbf-4ab9-9148-7f52e369b32a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:04:51.840563 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840544    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/522c68ff-5517-41f0-b9a0-2759155ac29f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "522c68ff-5517-41f0-b9a0-2759155ac29f" (UID: "522c68ff-5517-41f0-b9a0-2759155ac29f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:04:51.840682 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840635    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/522c68ff-5517-41f0-b9a0-2759155ac29f-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:51.840682 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.840654    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:51.842385 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.842355    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b829c2ab-4cbf-4ab9-9148-7f52e369b32a" (UID: "b829c2ab-4cbf-4ab9-9148-7f52e369b32a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:04:51.842488 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.842359    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-kube-api-access-8zlcm" (OuterVolumeSpecName: "kube-api-access-8zlcm") pod "b829c2ab-4cbf-4ab9-9148-7f52e369b32a" (UID: "b829c2ab-4cbf-4ab9-9148-7f52e369b32a"). InnerVolumeSpecName "kube-api-access-8zlcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:51.842724 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.842702    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b829c2ab-4cbf-4ab9-9148-7f52e369b32a" (UID: "b829c2ab-4cbf-4ab9-9148-7f52e369b32a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:04:51.842817 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.842722    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522c68ff-5517-41f0-b9a0-2759155ac29f-kube-api-access-x7d9d" (OuterVolumeSpecName: "kube-api-access-x7d9d") pod "522c68ff-5517-41f0-b9a0-2759155ac29f" (UID: "522c68ff-5517-41f0-b9a0-2759155ac29f"). InnerVolumeSpecName "kube-api-access-x7d9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:51.855329 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.855307    2563 generic.go:356] "Generic (PLEG): container finished" podID="522c68ff-5517-41f0-b9a0-2759155ac29f" containerID="0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b" exitCode=0
Sep 10 15:04:51.855429 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.855348    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"
Sep 10 15:04:51.855429 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.855382    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp" event={"ID":"522c68ff-5517-41f0-b9a0-2759155ac29f","Type":"ContainerDied","Data":"0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b"}
Sep 10 15:04:51.855429 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.855428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp" event={"ID":"522c68ff-5517-41f0-b9a0-2759155ac29f","Type":"ContainerDied","Data":"e2c43ff65948b13712d5808e4d33ceb0bbb01e44500b1e6762c9e4bea507e797"}
Sep 10 15:04:51.855596 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.855453    2563 scope.go:117] "RemoveContainer" containerID="0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b"
Sep 10 15:04:51.856813 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.856791    2563 generic.go:356] "Generic (PLEG): container finished" podID="b829c2ab-4cbf-4ab9-9148-7f52e369b32a" containerID="2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1" exitCode=0
Sep 10 15:04:51.856935 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.856821    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"
Sep 10 15:04:51.856935 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.856836    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh" event={"ID":"b829c2ab-4cbf-4ab9-9148-7f52e369b32a","Type":"ContainerDied","Data":"2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1"}
Sep 10 15:04:51.856935 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.856873    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh" event={"ID":"b829c2ab-4cbf-4ab9-9148-7f52e369b32a","Type":"ContainerDied","Data":"267389aafcc69c0dd9ded53ce7f8657dfcb7d0599a8a68579188df877098010d"}
Sep 10 15:04:51.865601 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.865585    2563 scope.go:117] "RemoveContainer" containerID="0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b"
Sep 10 15:04:51.865906 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:04:51.865886    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b\": container with ID starting with 0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b not found: ID does not exist" containerID="0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b"
Sep 10 15:04:51.865966 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.865915    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b"} err="failed to get container status \"0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b\": rpc error: code = NotFound desc = could not find container \"0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b\": container with ID starting with 0a69f153695c070cb7019319b50941979f2764c0365494a020fe4364c172bc5b not found: ID does not exist"
Sep 10 15:04:51.865966 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.865936    2563 scope.go:117] "RemoveContainer" containerID="2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1"
Sep 10 15:04:51.874630 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.874610    2563 scope.go:117] "RemoveContainer" containerID="2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1"
Sep 10 15:04:51.874922 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:04:51.874903    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1\": container with ID starting with 2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1 not found: ID does not exist" containerID="2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1"
Sep 10 15:04:51.874991 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.874928    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1"} err="failed to get container status \"2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1\": rpc error: code = NotFound desc = could not find container \"2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1\": container with ID starting with 2f46ad9fdec2f8a4b2bd4dc2e55f5f88f1d2f6f55c92ae0e2f2d644cf9a7e6c1 not found: ID does not exist"
Sep 10 15:04:51.878961 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.878941    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"]
Sep 10 15:04:51.883358 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.883338    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb65dc-66659598fc-q95zh"]
Sep 10 15:04:51.892597 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.892572    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"]
Sep 10 15:04:51.896545 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.896527    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb65dc-55cd7bc8ff-kc9vp"]
Sep 10 15:04:51.941879 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.941803    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:51.941879 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.941836    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:51.941879 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.941844    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zlcm\" (UniqueName: \"kubernetes.io/projected/b829c2ab-4cbf-4ab9-9148-7f52e369b32a-kube-api-access-8zlcm\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:51.941879 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.941854    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7d9d\" (UniqueName: \"kubernetes.io/projected/522c68ff-5517-41f0-b9a0-2759155ac29f-kube-api-access-x7d9d\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:51.984561 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:51.984487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:52.031881 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:52.031853    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522c68ff-5517-41f0-b9a0-2759155ac29f" path="/var/lib/kubelet/pods/522c68ff-5517-41f0-b9a0-2759155ac29f/volumes"
Sep 10 15:04:52.032230 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:52.032218    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b829c2ab-4cbf-4ab9-9148-7f52e369b32a" path="/var/lib/kubelet/pods/b829c2ab-4cbf-4ab9-9148-7f52e369b32a/volumes"
Sep 10 15:04:53.470114 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470082    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7659-12e321"]
Sep 10 15:04:53.470538 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470521    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522c68ff-5517-41f0-b9a0-2759155ac29f" containerName="payload-processing"
Sep 10 15:04:53.470538 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470533    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="522c68ff-5517-41f0-b9a0-2759155ac29f" containerName="payload-processing"
Sep 10 15:04:53.470632 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470552    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy"
Sep 10 15:04:53.470632 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470558    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy"
Sep 10 15:04:53.470632 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470566    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b829c2ab-4cbf-4ab9-9148-7f52e369b32a" containerName="maas-api"
Sep 10 15:04:53.470632 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470571    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b829c2ab-4cbf-4ab9-9148-7f52e369b32a" containerName="maas-api"
Sep 10 15:04:53.470632 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470580    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy"
Sep 10 15:04:53.470632 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470585    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy"
Sep 10 15:04:53.470840 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470646    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy"
Sep 10 15:04:53.470840 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470653    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cb9000-2ebf-4164-8d3e-1cfae086420b" containerName="istio-proxy"
Sep 10 15:04:53.470840 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470661    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="522c68ff-5517-41f0-b9a0-2759155ac29f" containerName="payload-processing"
Sep 10 15:04:53.470840 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.470669    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b829c2ab-4cbf-4ab9-9148-7f52e369b32a" containerName="maas-api"
Sep 10 15:04:53.475198 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.475182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-12e321"
Sep 10 15:04:53.477804 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.477630    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mmqf2\""
Sep 10 15:04:53.479663 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.479643    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-12e321"]
Sep 10 15:04:53.557910 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.557881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ksg\" (UniqueName: \"kubernetes.io/projected/66428439-991f-44b8-acc9-246cd3c5cd7c-kube-api-access-p4ksg\") pod \"test-curl-7659-12e321\" (UID: \"66428439-991f-44b8-acc9-246cd3c5cd7c\") " pod="opendatahub/test-curl-7659-12e321"
Sep 10 15:04:53.658388 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.658350    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ksg\" (UniqueName: \"kubernetes.io/projected/66428439-991f-44b8-acc9-246cd3c5cd7c-kube-api-access-p4ksg\") pod \"test-curl-7659-12e321\" (UID: \"66428439-991f-44b8-acc9-246cd3c5cd7c\") " pod="opendatahub/test-curl-7659-12e321"
Sep 10 15:04:53.666904 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.666879    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ksg\" (UniqueName: \"kubernetes.io/projected/66428439-991f-44b8-acc9-246cd3c5cd7c-kube-api-access-p4ksg\") pod \"test-curl-7659-12e321\" (UID: \"66428439-991f-44b8-acc9-246cd3c5cd7c\") " pod="opendatahub/test-curl-7659-12e321"
Sep 10 15:04:53.785957 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:53.785857    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-12e321"
Sep 10 15:04:54.058633 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:54.058612    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7659-12e321"]
Sep 10 15:04:54.060945 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:04:54.060919    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66428439_991f_44b8_acc9_246cd3c5cd7c.slice/crio-b1bd89cf4d83c53828c1b14b4eab6b66a3bd7cc27956b77615cd689d50b13f7a WatchSource:0}: Error finding container b1bd89cf4d83c53828c1b14b4eab6b66a3bd7cc27956b77615cd689d50b13f7a: Status 404 returned error can't find the container with id b1bd89cf4d83c53828c1b14b4eab6b66a3bd7cc27956b77615cd689d50b13f7a
Sep 10 15:04:54.878387 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:54.878342    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-12e321" event={"ID":"66428439-991f-44b8-acc9-246cd3c5cd7c","Type":"ContainerStarted","Data":"8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae"}
Sep 10 15:04:54.878769 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:54.878393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7659-12e321" event={"ID":"66428439-991f-44b8-acc9-246cd3c5cd7c","Type":"ContainerStarted","Data":"b1bd89cf4d83c53828c1b14b4eab6b66a3bd7cc27956b77615cd689d50b13f7a"}
Sep 10 15:04:54.892960 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:54.892917    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7659-12e321" podStartSLOduration=1.892901707 podStartE2EDuration="1.892901707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:04:54.892119479 +0000 UTC m=+1365.369179419" watchObservedRunningTime="2026-09-10 15:04:54.892901707 +0000 UTC m=+1365.369961651"
Sep 10 15:04:55.480476 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:55.480444    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7659-12e321"]
Sep 10 15:04:55.484571 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:55.484547    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7659-12e321"]
Sep 10 15:04:55.884097 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:04:55.884069    2563 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7659-12e321" container="test-curl-7659-12e321"
Sep 10 15:04:56.890580 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:56.890546    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7659-12e321" podUID="66428439-991f-44b8-acc9-246cd3c5cd7c" containerName="test-curl-7659-12e321" containerID="cri-o://8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae" gracePeriod=2
Sep 10 15:04:57.887119 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:57.887078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:57.965597 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:57.965563    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:58.134716 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:58.134672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:58.589773 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:58.589737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:59.070925 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.070904    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-12e321"
Sep 10 15:04:59.189111 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.188999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:04:59.202054 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.202026    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4ksg\" (UniqueName: \"kubernetes.io/projected/66428439-991f-44b8-acc9-246cd3c5cd7c-kube-api-access-p4ksg\") pod \"66428439-991f-44b8-acc9-246cd3c5cd7c\" (UID: \"66428439-991f-44b8-acc9-246cd3c5cd7c\") "
Sep 10 15:04:59.204126 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.204102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66428439-991f-44b8-acc9-246cd3c5cd7c-kube-api-access-p4ksg" (OuterVolumeSpecName: "kube-api-access-p4ksg") pod "66428439-991f-44b8-acc9-246cd3c5cd7c" (UID: "66428439-991f-44b8-acc9-246cd3c5cd7c"). InnerVolumeSpecName "kube-api-access-p4ksg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:04:59.302836 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.302805    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4ksg\" (UniqueName: \"kubernetes.io/projected/66428439-991f-44b8-acc9-246cd3c5cd7c-kube-api-access-p4ksg\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:04:59.909296 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.909256    2563 generic.go:356] "Generic (PLEG): container finished" podID="66428439-991f-44b8-acc9-246cd3c5cd7c" containerID="8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae" exitCode=137
Sep 10 15:04:59.909474 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.909309    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7659-12e321"
Sep 10 15:04:59.909474 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.909351    2563 scope.go:117] "RemoveContainer" containerID="8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae"
Sep 10 15:04:59.920451 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.920432    2563 scope.go:117] "RemoveContainer" containerID="8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae"
Sep 10 15:04:59.920778 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:04:59.920759    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae\": container with ID starting with 8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae not found: ID does not exist" containerID="8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae"
Sep 10 15:04:59.920838 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:04:59.920789    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae"} err="failed to get container status \"8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae\": rpc error: code = NotFound desc = could not find container \"8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae\": container with ID starting with 8987a176da7329d253fd524f6a740fa26524ad321d85d36419821084a26a5cae not found: ID does not exist"
Sep 10 15:04:59.921155 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:04:59.921133    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7659-12e321\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="66428439-991f-44b8-acc9-246cd3c5cd7c" pod="opendatahub/test-curl-7659-12e321"
Sep 10 15:05:00.029705 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:05:00.029661    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7659-12e321\" is forbidden: User \"system:node:ip-10-0-141-186.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-186.ec2.internal' and this object" podUID="66428439-991f-44b8-acc9-246cd3c5cd7c" pod="opendatahub/test-curl-7659-12e321"
Sep 10 15:05:00.032637 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:00.032610    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66428439-991f-44b8-acc9-246cd3c5cd7c" path="/var/lib/kubelet/pods/66428439-991f-44b8-acc9-246cd3c5cd7c/volumes"
Sep 10 15:05:00.103128 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:00.103097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:08.609316 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:08.609276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:09.192282 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.192246    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"]
Sep 10 15:05:09.192581 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.192478    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk" podUID="7a704f10-6b7d-489c-a890-82daa2a4be52" containerName="maas-api" containerID="cri-o://9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f" gracePeriod=30
Sep 10 15:05:09.254110 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.254071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:09.261402 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.261371    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"]
Sep 10 15:05:09.261669 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.261640    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z" podUID="8ba2dd72-5250-4fba-b733-71e3f4798e09" containerName="payload-processing" containerID="cri-o://f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e" gracePeriod=30
Sep 10 15:05:09.266587 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.266562    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"]
Sep 10 15:05:09.266998 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.266829    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg" podUID="979d03d3-e3c3-4f22-bfd9-8811e22c2734" containerName="payload-pre-processing" containerID="cri-o://f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce" gracePeriod=30
Sep 10 15:05:09.354712 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.354679    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:09.442359 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.442230    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:09.509784 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.509647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:09.528179 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.528151    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 15:05:09.557837 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.557809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 15:05:09.586280 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.586254    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 15:05:09.599361 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.599333    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc2jp\" (UniqueName: \"kubernetes.io/projected/8ba2dd72-5250-4fba-b733-71e3f4798e09-kube-api-access-tc2jp\") pod \"8ba2dd72-5250-4fba-b733-71e3f4798e09\" (UID: \"8ba2dd72-5250-4fba-b733-71e3f4798e09\") "
Sep 10 15:05:09.599536 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.599384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-maas-api-tls\") pod \"7a704f10-6b7d-489c-a890-82daa2a4be52\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") "
Sep 10 15:05:09.599536 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.599418    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a704f10-6b7d-489c-a890-82daa2a4be52-openshift-service-ca\") pod \"7a704f10-6b7d-489c-a890-82daa2a4be52\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") "
Sep 10 15:05:09.599536 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.599462    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-metrics-certs\") pod \"7a704f10-6b7d-489c-a890-82daa2a4be52\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") "
Sep 10 15:05:09.599536 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.599479    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba2dd72-5250-4fba-b733-71e3f4798e09-plugins-config-volume\") pod \"8ba2dd72-5250-4fba-b733-71e3f4798e09\" (UID: \"8ba2dd72-5250-4fba-b733-71e3f4798e09\") "
Sep 10 15:05:09.599744 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.599573    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62wl4\" (UniqueName: \"kubernetes.io/projected/7a704f10-6b7d-489c-a890-82daa2a4be52-kube-api-access-62wl4\") pod \"7a704f10-6b7d-489c-a890-82daa2a4be52\" (UID: \"7a704f10-6b7d-489c-a890-82daa2a4be52\") "
Sep 10 15:05:09.599880 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.599842    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a704f10-6b7d-489c-a890-82daa2a4be52-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a704f10-6b7d-489c-a890-82daa2a4be52" (UID: "7a704f10-6b7d-489c-a890-82daa2a4be52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:05:09.600016 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.599944    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba2dd72-5250-4fba-b733-71e3f4798e09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8ba2dd72-5250-4fba-b733-71e3f4798e09" (UID: "8ba2dd72-5250-4fba-b733-71e3f4798e09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:05:09.601685 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.601658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba2dd72-5250-4fba-b733-71e3f4798e09-kube-api-access-tc2jp" (OuterVolumeSpecName: "kube-api-access-tc2jp") pod "8ba2dd72-5250-4fba-b733-71e3f4798e09" (UID: "8ba2dd72-5250-4fba-b733-71e3f4798e09"). InnerVolumeSpecName "kube-api-access-tc2jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:05:09.601900 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.601765    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a704f10-6b7d-489c-a890-82daa2a4be52" (UID: "7a704f10-6b7d-489c-a890-82daa2a4be52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:05:09.601900 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.601854    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a704f10-6b7d-489c-a890-82daa2a4be52" (UID: "7a704f10-6b7d-489c-a890-82daa2a4be52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:05:09.602084 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.601918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a704f10-6b7d-489c-a890-82daa2a4be52-kube-api-access-62wl4" (OuterVolumeSpecName: "kube-api-access-62wl4") pod "7a704f10-6b7d-489c-a890-82daa2a4be52" (UID: "7a704f10-6b7d-489c-a890-82daa2a4be52"). InnerVolumeSpecName "kube-api-access-62wl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:05:09.700348 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.700253    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq5kv\" (UniqueName: \"kubernetes.io/projected/979d03d3-e3c3-4f22-bfd9-8811e22c2734-kube-api-access-jq5kv\") pod \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\" (UID: \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\") "
Sep 10 15:05:09.700348 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.700308    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979d03d3-e3c3-4f22-bfd9-8811e22c2734-plugins-config-volume\") pod \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\" (UID: \"979d03d3-e3c3-4f22-bfd9-8811e22c2734\") "
Sep 10 15:05:09.700843 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.700606    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:09.700843 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.700627    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a704f10-6b7d-489c-a890-82daa2a4be52-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:09.700843 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.700644    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a704f10-6b7d-489c-a890-82daa2a4be52-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:09.700843 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.700657    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ba2dd72-5250-4fba-b733-71e3f4798e09-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:09.700843 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.700669    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62wl4\" (UniqueName: \"kubernetes.io/projected/7a704f10-6b7d-489c-a890-82daa2a4be52-kube-api-access-62wl4\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:09.700843 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.700682    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc2jp\" (UniqueName: \"kubernetes.io/projected/8ba2dd72-5250-4fba-b733-71e3f4798e09-kube-api-access-tc2jp\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:09.700843 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.700707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979d03d3-e3c3-4f22-bfd9-8811e22c2734-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "979d03d3-e3c3-4f22-bfd9-8811e22c2734" (UID: "979d03d3-e3c3-4f22-bfd9-8811e22c2734"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:05:09.702354 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.702325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979d03d3-e3c3-4f22-bfd9-8811e22c2734-kube-api-access-jq5kv" (OuterVolumeSpecName: "kube-api-access-jq5kv") pod "979d03d3-e3c3-4f22-bfd9-8811e22c2734" (UID: "979d03d3-e3c3-4f22-bfd9-8811e22c2734"). InnerVolumeSpecName "kube-api-access-jq5kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:05:09.802026 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.801990    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq5kv\" (UniqueName: \"kubernetes.io/projected/979d03d3-e3c3-4f22-bfd9-8811e22c2734-kube-api-access-jq5kv\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:09.802026 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.802019    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/979d03d3-e3c3-4f22-bfd9-8811e22c2734-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:09.811301 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.811269    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:09.964307 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.964210    2563 generic.go:356] "Generic (PLEG): container finished" podID="8ba2dd72-5250-4fba-b733-71e3f4798e09" containerID="f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e" exitCode=0
Sep 10 15:05:09.964307 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.964268    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"
Sep 10 15:05:09.964307 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.964283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z" event={"ID":"8ba2dd72-5250-4fba-b733-71e3f4798e09","Type":"ContainerDied","Data":"f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e"}
Sep 10 15:05:09.964646 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.964358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z" event={"ID":"8ba2dd72-5250-4fba-b733-71e3f4798e09","Type":"ContainerDied","Data":"b08813bb8d7d95c437bab88cca2a0934c5dd3149deb6b51ab1c81ecb3177145e"}
Sep 10 15:05:09.964646 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.964387    2563 scope.go:117] "RemoveContainer" containerID="f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e"
Sep 10 15:05:09.965650 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.965628    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a704f10-6b7d-489c-a890-82daa2a4be52" containerID="9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f" exitCode=0
Sep 10 15:05:09.965754 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.965704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk" event={"ID":"7a704f10-6b7d-489c-a890-82daa2a4be52","Type":"ContainerDied","Data":"9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f"}
Sep 10 15:05:09.965754 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.965742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk" event={"ID":"7a704f10-6b7d-489c-a890-82daa2a4be52","Type":"ContainerDied","Data":"612c378a3213b32a4c3bb193c9b8c245e95986bc90410e3256f9c2ce4386f6aa"}
Sep 10 15:05:09.965876 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.965767    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"
Sep 10 15:05:09.967358 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.967332    2563 generic.go:356] "Generic (PLEG): container finished" podID="979d03d3-e3c3-4f22-bfd9-8811e22c2734" containerID="f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce" exitCode=0
Sep 10 15:05:09.967469 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.967394    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"
Sep 10 15:05:09.967564 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.967391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg" event={"ID":"979d03d3-e3c3-4f22-bfd9-8811e22c2734","Type":"ContainerDied","Data":"f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce"}
Sep 10 15:05:09.967564 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.967554    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg" event={"ID":"979d03d3-e3c3-4f22-bfd9-8811e22c2734","Type":"ContainerDied","Data":"5b66208a51f09c9b11d82d95d56db1fa5dcef262bf7d389a1d5e1523bf88b951"}
Sep 10 15:05:09.975186 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.975159    2563 scope.go:117] "RemoveContainer" containerID="f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e"
Sep 10 15:05:09.975537 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:05:09.975485    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e\": container with ID starting with f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e not found: ID does not exist" containerID="f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e"
Sep 10 15:05:09.975537 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.975524    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e"} err="failed to get container status \"f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e\": rpc error: code = NotFound desc = could not find container \"f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e\": container with ID starting with f006e7f6b14bf307162a3b31bfff0371104505c786e26f37eff0f4408c64d06e not found: ID does not exist"
Sep 10 15:05:09.975693 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.975545    2563 scope.go:117] "RemoveContainer" containerID="9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f"
Sep 10 15:05:09.984271 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.984250    2563 scope.go:117] "RemoveContainer" containerID="9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f"
Sep 10 15:05:09.984715 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:05:09.984688    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f\": container with ID starting with 9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f not found: ID does not exist" containerID="9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f"
Sep 10 15:05:09.984797 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.984722    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f"} err="failed to get container status \"9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f\": rpc error: code = NotFound desc = could not find container \"9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f\": container with ID starting with 9d0100e493bd3ed5b848db3c1428e4ae338aadc17c2e1bc7fbdace2df4a8802f not found: ID does not exist"
Sep 10 15:05:09.984797 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.984740    2563 scope.go:117] "RemoveContainer" containerID="f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce"
Sep 10 15:05:09.987109 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.987088    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"]
Sep 10 15:05:09.992124 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.992103    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5fb994-767b59d97f-59l8z"]
Sep 10 15:05:09.993645 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.993632    2563 scope.go:117] "RemoveContainer" containerID="f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce"
Sep 10 15:05:09.993869 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:05:09.993853    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce\": container with ID starting with f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce not found: ID does not exist" containerID="f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce"
Sep 10 15:05:09.993913 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:09.993875    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce"} err="failed to get container status \"f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce\": rpc error: code = NotFound desc = could not find container \"f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce\": container with ID starting with f474a3033331d25e56acc27223962b2480fef27058674cafac2df7c9030c69ce not found: ID does not exist"
Sep 10 15:05:10.001386 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:10.001365    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"]
Sep 10 15:05:10.005787 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:10.005768    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5fb994-6cb7f9458b-24dbg"]
Sep 10 15:05:10.015345 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:10.015323    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"]
Sep 10 15:05:10.017322 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:10.017300    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5fb994-7f5685dc67-nmsqk"]
Sep 10 15:05:10.031924 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:10.031896    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a704f10-6b7d-489c-a890-82daa2a4be52" path="/var/lib/kubelet/pods/7a704f10-6b7d-489c-a890-82daa2a4be52/volumes"
Sep 10 15:05:10.032266 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:10.032253    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba2dd72-5250-4fba-b733-71e3f4798e09" path="/var/lib/kubelet/pods/8ba2dd72-5250-4fba-b733-71e3f4798e09/volumes"
Sep 10 15:05:10.032595 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:10.032582    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="979d03d3-e3c3-4f22-bfd9-8811e22c2734" path="/var/lib/kubelet/pods/979d03d3-e3c3-4f22-bfd9-8811e22c2734/volumes"
Sep 10 15:05:15.605440 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:15.605401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:15.670151 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:15.670108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:15.880228 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:15.880141    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:16.446100 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:16.446045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:23.963618 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:23.963558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:26.857751 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:26.857711    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"]
Sep 10 15:05:26.858175 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:26.857931    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69" podUID="32e70000-fc43-4ad5-b7f7-75c735a2b9c3" containerName="maas-api" containerID="cri-o://c490608b2ee4cba5713933ea68cc6a4629e7007c7687b235b8c5218cb0b378fd" gracePeriod=30
Sep 10 15:05:26.990827 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:26.990765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:27.011257 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.011189    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69" podUID="32e70000-fc43-4ad5-b7f7-75c735a2b9c3" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.59:8443/health\": dial tcp 10.132.0.59:8443: connect: connection refused"
Sep 10 15:05:27.068078 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.068044    2563 generic.go:356] "Generic (PLEG): container finished" podID="32e70000-fc43-4ad5-b7f7-75c735a2b9c3" containerID="c490608b2ee4cba5713933ea68cc6a4629e7007c7687b235b8c5218cb0b378fd" exitCode=0
Sep 10 15:05:27.068251 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.068118    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69" event={"ID":"32e70000-fc43-4ad5-b7f7-75c735a2b9c3","Type":"ContainerDied","Data":"c490608b2ee4cba5713933ea68cc6a4629e7007c7687b235b8c5218cb0b378fd"}
Sep 10 15:05:27.093129 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.093097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:27.116721 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.116655    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 15:05:27.195935 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.195893    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:27.266161 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.266130    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-metrics-certs\") pod \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") "
Sep 10 15:05:27.266315 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.266232    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-openshift-service-ca\") pod \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") "
Sep 10 15:05:27.266426 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.266406    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgvpv\" (UniqueName: \"kubernetes.io/projected/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-kube-api-access-jgvpv\") pod \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") "
Sep 10 15:05:27.266490 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.266478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-maas-api-tls\") pod \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\" (UID: \"32e70000-fc43-4ad5-b7f7-75c735a2b9c3\") "
Sep 10 15:05:27.266585 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.266556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32e70000-fc43-4ad5-b7f7-75c735a2b9c3" (UID: "32e70000-fc43-4ad5-b7f7-75c735a2b9c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:05:27.266845 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.266820    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:27.268477 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.268444    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32e70000-fc43-4ad5-b7f7-75c735a2b9c3" (UID: "32e70000-fc43-4ad5-b7f7-75c735a2b9c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:05:27.268620 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.268478    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32e70000-fc43-4ad5-b7f7-75c735a2b9c3" (UID: "32e70000-fc43-4ad5-b7f7-75c735a2b9c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:05:27.268620 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.268500    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-kube-api-access-jgvpv" (OuterVolumeSpecName: "kube-api-access-jgvpv") pod "32e70000-fc43-4ad5-b7f7-75c735a2b9c3" (UID: "32e70000-fc43-4ad5-b7f7-75c735a2b9c3"). InnerVolumeSpecName "kube-api-access-jgvpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:05:27.368239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.368159    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:27.368239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.368186    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:27.368239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.368197    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgvpv\" (UniqueName: \"kubernetes.io/projected/32e70000-fc43-4ad5-b7f7-75c735a2b9c3-kube-api-access-jgvpv\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:27.488906 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:27.488869    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:28.074734 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:28.074701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69" event={"ID":"32e70000-fc43-4ad5-b7f7-75c735a2b9c3","Type":"ContainerDied","Data":"2328f24baf7f3bbadbbf725fda2550cd2b978f471d5562222fbfe99a8264450c"}
Sep 10 15:05:28.075100 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:28.074762    2563 scope.go:117] "RemoveContainer" containerID="c490608b2ee4cba5713933ea68cc6a4629e7007c7687b235b8c5218cb0b378fd"
Sep 10 15:05:28.075100 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:28.074716    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"
Sep 10 15:05:28.090609 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:28.090584    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"]
Sep 10 15:05:28.094514 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:28.094483    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5899e8-8575cfc999-66d69"]
Sep 10 15:05:29.325089 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:29.325058    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:30.032867 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:30.032826    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32e70000-fc43-4ad5-b7f7-75c735a2b9c3" path="/var/lib/kubelet/pods/32e70000-fc43-4ad5-b7f7-75c735a2b9c3/volumes"
Sep 10 15:05:33.895796 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:33.895763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:33.959597 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:33.959544    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:34.130901 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:34.130867    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"]
Sep 10 15:05:34.131154 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:34.131104    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy" containerID="cri-o://cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5" gracePeriod=30
Sep 10 15:05:34.215854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:34.215772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:36.399231 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:36.399197    2563 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:05:36.399604 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:36.399250    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:05:39.391102 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.391079    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 15:05:39.482824 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.482732    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-data\") pod \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") "
Sep 10 15:05:39.482824 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.482773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istiod-ca-cert\") pod \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") "
Sep 10 15:05:39.482824 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.482799    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-token\") pod \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") "
Sep 10 15:05:39.483107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.482827    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-credential-socket\") pod \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") "
Sep 10 15:05:39.483107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.482960    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-socket\") pod \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") "
Sep 10 15:05:39.483107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483022    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-podinfo\") pod \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") "
Sep 10 15:05:39.483107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483031    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-data" (OuterVolumeSpecName: "istio-data") pod "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" (UID: "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:39.483107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9x4tx\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-kube-api-access-9x4tx\") pod \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") "
Sep 10 15:05:39.483107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" (UID: "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:39.483107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483100    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-envoy\") pod \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") "
Sep 10 15:05:39.483450 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483153    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-certs\") pod \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\" (UID: \"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f\") "
Sep 10 15:05:39.483450 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" (UID: "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:39.483450 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483217    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" (UID: "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:05:39.483635 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483470    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" (UID: "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:39.483635 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483602    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:39.483635 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483625    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-data\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:39.483774 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483637    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istiod-ca-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:39.483774 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483651    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-credential-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:39.483774 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.483665    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-workload-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:39.485090 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.485069    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-token" (OuterVolumeSpecName: "istio-token") pod "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" (UID: "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:05:39.485189 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.485145    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" (UID: "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:05:39.485443 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.485423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" (UID: "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:05:39.485565 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.485441    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-kube-api-access-9x4tx" (OuterVolumeSpecName: "kube-api-access-9x4tx") pod "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" (UID: "da0c2c19-7193-44f8-b1fc-d3c7bc644b6f"). InnerVolumeSpecName "kube-api-access-9x4tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:05:39.584787 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.584756    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-token\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:39.584787 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.584783    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-podinfo\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:39.584787 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.584792    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x4tx\" (UniqueName: \"kubernetes.io/projected/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-kube-api-access-9x4tx\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:39.585019 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:39.584802    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f-istio-envoy\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:05:40.135661 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.135618    2563 generic.go:356] "Generic (PLEG): container finished" podID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerID="cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5" exitCode=0
Sep 10 15:05:40.135826 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.135690    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"
Sep 10 15:05:40.135826 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.135702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" event={"ID":"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f","Type":"ContainerDied","Data":"cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5"}
Sep 10 15:05:40.135826 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.135774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9" event={"ID":"da0c2c19-7193-44f8-b1fc-d3c7bc644b6f","Type":"ContainerDied","Data":"42e1a2ac9e942f7ffb20cfaca6adb638d02c18b31812b797b58dc8a62eb49c3f"}
Sep 10 15:05:40.135826 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.135805    2563 scope.go:117] "RemoveContainer" containerID="cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5"
Sep 10 15:05:40.144637 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.144618    2563 scope.go:117] "RemoveContainer" containerID="f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf"
Sep 10 15:05:40.153392 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.153370    2563 scope.go:117] "RemoveContainer" containerID="cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5"
Sep 10 15:05:40.153732 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:05:40.153712    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5\": container with ID starting with cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5 not found: ID does not exist" containerID="cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5"
Sep 10 15:05:40.153836 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.153739    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5"} err="failed to get container status \"cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5\": rpc error: code = NotFound desc = could not find container \"cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5\": container with ID starting with cf8a1abc7323967480335a21c19fbca1e841abdb71c4f0ddc74e22c3b4cc1ad5 not found: ID does not exist"
Sep 10 15:05:40.153836 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.153756    2563 scope.go:117] "RemoveContainer" containerID="f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf"
Sep 10 15:05:40.154064 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:05:40.154046    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf\": container with ID starting with f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf not found: ID does not exist" containerID="f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf"
Sep 10 15:05:40.154101 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.154071    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf"} err="failed to get container status \"f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf\": rpc error: code = NotFound desc = could not find container \"f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf\": container with ID starting with f4ddea3cb1f4f48c49c9a32291326e691ba56775b1ad4547f4d15f7c45e209bf not found: ID does not exist"
Sep 10 15:05:40.155355 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.155338    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"]
Sep 10 15:05:40.163074 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.163054    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5899e8-openshift-default-59f95568dd-8qfg9"]
Sep 10 15:05:40.184516 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:40.184471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:05:42.032220 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:05:42.032186    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" path="/var/lib/kubelet/pods/da0c2c19-7193-44f8-b1fc-d3c7bc644b6f/volumes"
Sep 10 15:06:03.377454 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377367    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"]
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377822    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="979d03d3-e3c3-4f22-bfd9-8811e22c2734" containerName="payload-pre-processing"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377836    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="979d03d3-e3c3-4f22-bfd9-8811e22c2734" containerName="payload-pre-processing"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377843    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377849    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377859    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e70000-fc43-4ad5-b7f7-75c735a2b9c3" containerName="maas-api"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377864    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e70000-fc43-4ad5-b7f7-75c735a2b9c3" containerName="maas-api"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377871    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66428439-991f-44b8-acc9-246cd3c5cd7c" containerName="test-curl-7659-12e321"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377876    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66428439-991f-44b8-acc9-246cd3c5cd7c" containerName="test-curl-7659-12e321"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377890    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba2dd72-5250-4fba-b733-71e3f4798e09" containerName="payload-processing"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377895    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba2dd72-5250-4fba-b733-71e3f4798e09" containerName="payload-processing"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377902    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a704f10-6b7d-489c-a890-82daa2a4be52" containerName="maas-api"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377907    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a704f10-6b7d-489c-a890-82daa2a4be52" containerName="maas-api"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377961    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e70000-fc43-4ad5-b7f7-75c735a2b9c3" containerName="maas-api"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377970    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a704f10-6b7d-489c-a890-82daa2a4be52" containerName="maas-api"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377975    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="979d03d3-e3c3-4f22-bfd9-8811e22c2734" containerName="payload-pre-processing"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377981    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba2dd72-5250-4fba-b733-71e3f4798e09" containerName="payload-processing"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377987    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377992    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="66428439-991f-44b8-acc9-246cd3c5cd7c" containerName="test-curl-7659-12e321"
Sep 10 15:06:03.378001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.377999    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy"
Sep 10 15:06:03.382769 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.382747    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.385065 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.385043    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f81b7\"/\"openshift-service-ca.crt\""
Sep 10 15:06:03.385666 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.385643    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f81b7\"/\"default-dockercfg-dfdcl\""
Sep 10 15:06:03.385770 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.385670    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f81b7\"/\"rate-test-model-3f81b7-kserve-self-signed-certs\""
Sep 10 15:06:03.385770 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.385678    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-3f81b7\"/\"kube-root-ca.crt\""
Sep 10 15:06:03.391614 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.391591    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"]
Sep 10 15:06:03.409499 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.409473    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:03.507742 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.507709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74qtr\" (UniqueName: \"kubernetes.io/projected/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-kube-api-access-74qtr\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.507919 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.507757    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-model-cache\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.507919 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.507845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-dshm\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.507919 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.507903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tmp-dir\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.508074 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.507930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tls-certs\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.508074 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.507964    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-home\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.609020 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.608986    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tmp-dir\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.609020 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.609025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tls-certs\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.609299 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.609084    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-home\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.609299 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.609116    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74qtr\" (UniqueName: \"kubernetes.io/projected/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-kube-api-access-74qtr\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.609299 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.609157    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-model-cache\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.609299 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.609210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-dshm\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.609638 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.609610    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-model-cache\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.609819 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.609795    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tmp-dir\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.609905 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.609825    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-home\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.611797 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.611771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-dshm\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.611918 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.611900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tls-certs\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.617384 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.617363    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74qtr\" (UniqueName: \"kubernetes.io/projected/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-kube-api-access-74qtr\") pod \"rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") " pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.693696 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.693666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:03.967596 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:03.967569    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"]
Sep 10 15:06:03.970318 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:06:03.970294    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4fa202f_f128_4cc5_9b6c_9d0f4897ef99.slice/crio-a1f44fa11d54ad05d314e5aff5ead34154d20e7c546cf55d8a5d5de42599b1b3 WatchSource:0}: Error finding container a1f44fa11d54ad05d314e5aff5ead34154d20e7c546cf55d8a5d5de42599b1b3: Status 404 returned error can't find the container with id a1f44fa11d54ad05d314e5aff5ead34154d20e7c546cf55d8a5d5de42599b1b3
Sep 10 15:06:04.268411 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:04.268319    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q" event={"ID":"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99","Type":"ContainerStarted","Data":"907d539a8aec4e8444e65b6c74799c1ee2c07720db1a57f68f38912c1abfc3d8"}
Sep 10 15:06:04.268411 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:04.268368    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q" event={"ID":"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99","Type":"ContainerStarted","Data":"a1f44fa11d54ad05d314e5aff5ead34154d20e7c546cf55d8a5d5de42599b1b3"}
Sep 10 15:06:04.285341 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:04.285294    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q" podStartSLOduration=1.285277227 podStartE2EDuration="1.285277227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:04.283398011 +0000 UTC m=+1434.760457946" watchObservedRunningTime="2026-09-10 15:06:04.285277227 +0000 UTC m=+1434.762337171"
Sep 10 15:06:13.694083 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:13.694044    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:13.694491 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:13.694097    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:13.706856 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:13.706831    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:14.330380 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:14.330351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:20.710955 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:20.710920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:20.813015 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:20.812977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:39.186691 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:39.186647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:39.259496 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:39.259470    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"]
Sep 10 15:06:39.272422 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:39.272389    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:39.729086 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:39.729053    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:40.019041 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.018963    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"]
Sep 10 15:06:40.019528 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.019498    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy"
Sep 10 15:06:40.019590 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.019532    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="da0c2c19-7193-44f8-b1fc-d3c7bc644b6f" containerName="istio-proxy"
Sep 10 15:06:40.024286 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.024267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.026697 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.026661    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:06:40.026871 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.026722    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b022ee-openshift-default-dockercfg-jxhtg\""
Sep 10 15:06:40.037591 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.037567    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"]
Sep 10 15:06:40.044526 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.044462    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:40.121600 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.121565    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-podinfo\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.121797 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.121623    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-kube-api-access-dmd6k\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.121797 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.121691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-envoy\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.121797 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.121730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-token\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.121797 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.121766    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998f4479-b372-4c83-b1da-e6c71bc16d70-istiod-ca-cert\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.121966 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.121832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-certs\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.121966 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.121867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-data\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.121966 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.121889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-socket\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.121966 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.121913    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-credential-socket\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.222809 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.222769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-credential-socket\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223203 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.222862    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-podinfo\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223203 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.222903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-kube-api-access-dmd6k\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223203 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.222930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-envoy\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223203 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.222954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-token\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223203 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.222990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998f4479-b372-4c83-b1da-e6c71bc16d70-istiod-ca-cert\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223203 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.223037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-certs\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223203 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.223072    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-data\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223203 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.223101    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-socket\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223203 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.223172    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-credential-socket\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223733 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.223424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-socket\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223733 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.223583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-data\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223859 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.223841    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-certs\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.223959 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.223936    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998f4479-b372-4c83-b1da-e6c71bc16d70-istiod-ca-cert\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.225319 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.225290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-envoy\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.225680 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.225656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-podinfo\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.234434 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.234409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-token\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.234877 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.234853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-kube-api-access-dmd6k\") pod \"e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") " pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.337306 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.337225    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:40.376154 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.376120    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:40.615042 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.615008    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"]
Sep 10 15:06:40.617024 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:06:40.616996    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998f4479_b372_4c83_b1da_e6c71bc16d70.slice/crio-4f56055ca7e3a9912a8ea7480e30f9ceaca2f3b9a55467441fc1d449e599fb1f WatchSource:0}: Error finding container 4f56055ca7e3a9912a8ea7480e30f9ceaca2f3b9a55467441fc1d449e599fb1f: Status 404 returned error can't find the container with id 4f56055ca7e3a9912a8ea7480e30f9ceaca2f3b9a55467441fc1d449e599fb1f
Sep 10 15:06:40.619885 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.619853    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:06:40.619998 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.619929    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:06:40.619998 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:40.619975    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:06:41.451883 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:41.451852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" event={"ID":"998f4479-b372-4c83-b1da-e6c71bc16d70","Type":"ContainerStarted","Data":"8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea"}
Sep 10 15:06:41.452286 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:41.451903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" event={"ID":"998f4479-b372-4c83-b1da-e6c71bc16d70","Type":"ContainerStarted","Data":"4f56055ca7e3a9912a8ea7480e30f9ceaca2f3b9a55467441fc1d449e599fb1f"}
Sep 10 15:06:41.473259 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:41.473210    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" podStartSLOduration=2.473193981 podStartE2EDuration="2.473193981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:41.472486719 +0000 UTC m=+1471.949546665" watchObservedRunningTime="2026-09-10 15:06:41.473193981 +0000 UTC m=+1471.950253925"
Sep 10 15:06:42.337385 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:42.337351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:42.342839 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:42.342809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:42.457143 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:42.457108    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:42.458225 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:42.458207    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:06:43.433245 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:43.433201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:43.520289 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:43.520255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:44.709238 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:44.709192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:44.816400 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:44.816362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:45.127133 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.127056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:45.244184 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.244144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:45.347004 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.346972    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:06:45.386449 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.386374    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"]
Sep 10 15:06:45.390093 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.390076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:45.392172 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.392151    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b022ee\""
Sep 10 15:06:45.400884 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.400862    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"]
Sep 10 15:06:45.449679 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.449646    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"]
Sep 10 15:06:45.453407 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.453383    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:45.456739 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.456719    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b022ee-dockercfg-cwpcq\""
Sep 10 15:06:45.464654 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.464635    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"]
Sep 10 15:06:45.468989 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.468969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8tcv\" (UniqueName: \"kubernetes.io/projected/7b612830-9f63-4cb5-8b50-19d06ffa789f-kube-api-access-f8tcv\") pod \"payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z\" (UID: \"7b612830-9f63-4cb5-8b50-19d06ffa789f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:45.469105 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.468995    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b612830-9f63-4cb5-8b50-19d06ffa789f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z\" (UID: \"7b612830-9f63-4cb5-8b50-19d06ffa789f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:45.469105 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.469055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edea59c-718b-4ce3-90a9-699f9f56aa68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx\" (UID: \"2edea59c-718b-4ce3-90a9-699f9f56aa68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:45.469258 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.469235    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmkb\" (UniqueName: \"kubernetes.io/projected/2edea59c-718b-4ce3-90a9-699f9f56aa68-kube-api-access-jgmkb\") pod \"payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx\" (UID: \"2edea59c-718b-4ce3-90a9-699f9f56aa68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:45.569852 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.569803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8tcv\" (UniqueName: \"kubernetes.io/projected/7b612830-9f63-4cb5-8b50-19d06ffa789f-kube-api-access-f8tcv\") pod \"payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z\" (UID: \"7b612830-9f63-4cb5-8b50-19d06ffa789f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:45.569852 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.569855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b612830-9f63-4cb5-8b50-19d06ffa789f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z\" (UID: \"7b612830-9f63-4cb5-8b50-19d06ffa789f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:45.570107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.569885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edea59c-718b-4ce3-90a9-699f9f56aa68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx\" (UID: \"2edea59c-718b-4ce3-90a9-699f9f56aa68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:45.570107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.570022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmkb\" (UniqueName: \"kubernetes.io/projected/2edea59c-718b-4ce3-90a9-699f9f56aa68-kube-api-access-jgmkb\") pod \"payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx\" (UID: \"2edea59c-718b-4ce3-90a9-699f9f56aa68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:45.570676 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.570650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edea59c-718b-4ce3-90a9-699f9f56aa68-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx\" (UID: \"2edea59c-718b-4ce3-90a9-699f9f56aa68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:45.570785 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.570656    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b612830-9f63-4cb5-8b50-19d06ffa789f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z\" (UID: \"7b612830-9f63-4cb5-8b50-19d06ffa789f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:45.578399 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.578379    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmkb\" (UniqueName: \"kubernetes.io/projected/2edea59c-718b-4ce3-90a9-699f9f56aa68-kube-api-access-jgmkb\") pod \"payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx\" (UID: \"2edea59c-718b-4ce3-90a9-699f9f56aa68\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:45.578615 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.578595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8tcv\" (UniqueName: \"kubernetes.io/projected/7b612830-9f63-4cb5-8b50-19d06ffa789f-kube-api-access-f8tcv\") pod \"payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z\" (UID: \"7b612830-9f63-4cb5-8b50-19d06ffa789f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:45.701280 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.701248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:45.763830 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.763790    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:45.982729 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:45.982706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"]
Sep 10 15:06:45.985690 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:06:45.985663    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edea59c_718b_4ce3_90a9_699f9f56aa68.slice/crio-65c1ec3f45c4136ab8ddf829e1738b5bbab75ecaac0c4e266bad9cc93a7b4a13 WatchSource:0}: Error finding container 65c1ec3f45c4136ab8ddf829e1738b5bbab75ecaac0c4e266bad9cc93a7b4a13: Status 404 returned error can't find the container with id 65c1ec3f45c4136ab8ddf829e1738b5bbab75ecaac0c4e266bad9cc93a7b4a13
Sep 10 15:06:46.050083 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:46.050060    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"]
Sep 10 15:06:46.054064 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:06:46.054042    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b612830_9f63_4cb5_8b50_19d06ffa789f.slice/crio-b6aa8c18c821be72070f97b3e10367cd3d39f64273bae120679044b0f361a159 WatchSource:0}: Error finding container b6aa8c18c821be72070f97b3e10367cd3d39f64273bae120679044b0f361a159: Status 404 returned error can't find the container with id b6aa8c18c821be72070f97b3e10367cd3d39f64273bae120679044b0f361a159
Sep 10 15:06:46.478262 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:46.478221    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z" event={"ID":"7b612830-9f63-4cb5-8b50-19d06ffa789f","Type":"ContainerStarted","Data":"405dffbd7282df0630e76a31275192d62ff3d8d9d76a8b2ac5900bf209f3e359"}
Sep 10 15:06:46.478456 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:46.478289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z" event={"ID":"7b612830-9f63-4cb5-8b50-19d06ffa789f","Type":"ContainerStarted","Data":"b6aa8c18c821be72070f97b3e10367cd3d39f64273bae120679044b0f361a159"}
Sep 10 15:06:46.478456 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:46.478317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:46.480026 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:46.479997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx" event={"ID":"2edea59c-718b-4ce3-90a9-699f9f56aa68","Type":"ContainerStarted","Data":"510bb0570f29f852d40d42081759ac237e2897bb8efa8623062cc453a91412b2"}
Sep 10 15:06:46.480158 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:46.480049    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx" event={"ID":"2edea59c-718b-4ce3-90a9-699f9f56aa68","Type":"ContainerStarted","Data":"65c1ec3f45c4136ab8ddf829e1738b5bbab75ecaac0c4e266bad9cc93a7b4a13"}
Sep 10 15:06:46.480158 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:46.480098    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:46.507881 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:46.507833    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z" podStartSLOduration=1.5078187760000001 podStartE2EDuration="1.507818776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:46.505723194 +0000 UTC m=+1476.982783141" watchObservedRunningTime="2026-09-10 15:06:46.507818776 +0000 UTC m=+1476.984878720"
Sep 10 15:06:46.529777 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:46.529717    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx" podStartSLOduration=1.52969635 podStartE2EDuration="1.52969635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:06:46.528048398 +0000 UTC m=+1477.005108342" watchObservedRunningTime="2026-09-10 15:06:46.52969635 +0000 UTC m=+1477.006756295"
Sep 10 15:06:54.278993 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.278953    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q" podUID="e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" containerName="main" containerID="cri-o://907d539a8aec4e8444e65b6c74799c1ee2c07720db1a57f68f38912c1abfc3d8" gracePeriod=45
Sep 10 15:06:54.320208 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.320157    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q" podUID="e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": dial tcp 10.132.0.68:8000: connect: connection refused"
Sep 10 15:06:54.549262 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.549170    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" containerID="907d539a8aec4e8444e65b6c74799c1ee2c07720db1a57f68f38912c1abfc3d8" exitCode=0
Sep 10 15:06:54.549262 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.549253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q" event={"ID":"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99","Type":"ContainerDied","Data":"907d539a8aec4e8444e65b6c74799c1ee2c07720db1a57f68f38912c1abfc3d8"}
Sep 10 15:06:54.580052 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.580026    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:54.677228 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677191    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tmp-dir\") pod \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") "
Sep 10 15:06:54.677397 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677284    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-dshm\") pod \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") "
Sep 10 15:06:54.677397 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677314    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-home\") pod \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") "
Sep 10 15:06:54.677397 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tls-certs\") pod \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") "
Sep 10 15:06:54.677577 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74qtr\" (UniqueName: \"kubernetes.io/projected/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-kube-api-access-74qtr\") pod \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") "
Sep 10 15:06:54.677577 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677430    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" (UID: "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:54.677577 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677437    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-model-cache\") pod \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\" (UID: \"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99\") "
Sep 10 15:06:54.677748 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-home" (OuterVolumeSpecName: "home") pod "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" (UID: "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:54.677748 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677708    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" (UID: "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:54.677858 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677802    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-home\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:06:54.677858 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677823    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-model-cache\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:06:54.677858 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.677838    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tmp-dir\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:06:54.679660 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.679634    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-kube-api-access-74qtr" (OuterVolumeSpecName: "kube-api-access-74qtr") pod "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" (UID: "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99"). InnerVolumeSpecName "kube-api-access-74qtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:06:54.679660 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.679648    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" (UID: "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:06:54.679787 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.679707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-dshm" (OuterVolumeSpecName: "dshm") pod "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" (UID: "e4fa202f-f128-4cc5-9b6c-9d0f4897ef99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:06:54.778330 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.778300    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-dshm\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:06:54.778330 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.778328    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-tls-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:06:54.778579 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:54.778339    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74qtr\" (UniqueName: \"kubernetes.io/projected/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99-kube-api-access-74qtr\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:06:55.555576 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:55.555549    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"
Sep 10 15:06:55.555992 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:55.555546    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q" event={"ID":"e4fa202f-f128-4cc5-9b6c-9d0f4897ef99","Type":"ContainerDied","Data":"a1f44fa11d54ad05d314e5aff5ead34154d20e7c546cf55d8a5d5de42599b1b3"}
Sep 10 15:06:55.555992 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:55.555667    2563 scope.go:117] "RemoveContainer" containerID="907d539a8aec4e8444e65b6c74799c1ee2c07720db1a57f68f38912c1abfc3d8"
Sep 10 15:06:55.576727 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:55.576699    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"]
Sep 10 15:06:55.580687 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:55.580662    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-3f81b7/rate-test-model-3f81b7-kserve-7bf8ddc8ff-vc52q"]
Sep 10 15:06:56.032673 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:56.032632    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" path="/var/lib/kubelet/pods/e4fa202f-f128-4cc5-9b6c-9d0f4897ef99/volumes"
Sep 10 15:06:57.487713 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:57.487683    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:06:57.488184 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:57.487851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:06:58.276131 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:58.276101    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": read tcp 10.132.0.2:55818->10.132.0.69:15021: read: connection reset by peer" start-of-body=
Sep 10 15:06:58.276289 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:58.276183    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": read tcp 10.132.0.2:55818->10.132.0.69:15021: read: connection reset by peer"
Sep 10 15:06:58.577879 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:58.577777    2563 generic.go:356] "Generic (PLEG): container finished" podID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerID="8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea" exitCode=0
Sep 10 15:06:58.577879 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:58.577831    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" event={"ID":"998f4479-b372-4c83-b1da-e6c71bc16d70","Type":"ContainerDied","Data":"8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea"}
Sep 10 15:06:58.578275 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:58.578232    2563 scope.go:117] "RemoveContainer" containerID="8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea"
Sep 10 15:06:58.579347 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:58.579316    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:06:58.579440 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:58.579393    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:06:58.579440 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:58.579424    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:06:59.592100 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:06:59.592066    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" event={"ID":"998f4479-b372-4c83-b1da-e6c71bc16d70","Type":"ContainerStarted","Data":"b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a"}
Sep 10 15:07:00.338390 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:00.338342    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:07:00.338642 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:00.338580    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:07:00.338642 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:00.338631    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 15:07:01.338297 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:01.338245    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:07:01.338830 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:01.338317    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 15:07:02.338285 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:02.338250    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:07:02.338475 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:02.338302    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 15:07:03.337995 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:03.337953    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:07:03.338185 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:03.338012    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 15:07:04.338437 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:04.338403    2563 patch_prober.go:28] interesting pod/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 10 15:07:04.338950 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:04.338467    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 10 15:07:05.341762 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:05.341734    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:07:05.342168 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:05.341994    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:07:05.342714 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:05.342694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:07:08.382404 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:08.382374    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:09.065488 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:09.065440    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:09.391107 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:09.391068    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:09.596012 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:09.595978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:10.069777 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:10.069753    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 15:07:10.079828 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:10.079804    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 15:07:12.632671 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:12.632634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:12.739341 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:12.739289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:13.776263 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.776229    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"]
Sep 10 15:07:13.776876 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.776856    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" containerName="main"
Sep 10 15:07:13.776963 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.776880    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" containerName="main"
Sep 10 15:07:13.777017 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.776985    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4fa202f-f128-4cc5-9b6c-9d0f4897ef99" containerName="main"
Sep 10 15:07:13.780267 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.780244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:13.782970 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.782949    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-cb5c8c-serving-cert\""
Sep 10 15:07:13.795953 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.795928    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"]
Sep 10 15:07:13.820833 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.820799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:13.903476 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.903437    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:13.964812 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.964776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6p7p\" (UniqueName: \"kubernetes.io/projected/71c5e681-023c-47a3-a857-4f4f51cdb7b2-kube-api-access-j6p7p\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:13.964812 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.964822    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c5e681-023c-47a3-a857-4f4f51cdb7b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:13.965042 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.964905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:13.965042 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:13.964935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.005607 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.005575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:14.066268 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.066178    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6p7p\" (UniqueName: \"kubernetes.io/projected/71c5e681-023c-47a3-a857-4f4f51cdb7b2-kube-api-access-j6p7p\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.066268 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.066219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c5e681-023c-47a3-a857-4f4f51cdb7b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.066530 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.066270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.066530 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.066297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.066908 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.066880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c5e681-023c-47a3-a857-4f4f51cdb7b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.068868 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.068845    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.068956 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.068908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.075120 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.075097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6p7p\" (UniqueName: \"kubernetes.io/projected/71c5e681-023c-47a3-a857-4f4f51cdb7b2-kube-api-access-j6p7p\") pod \"maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.090771 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.090746    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:14.371551 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.371284    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"]
Sep 10 15:07:14.374652 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:07:14.374614    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71c5e681_023c_47a3_a857_4f4f51cdb7b2.slice/crio-8ffb960804bab04507afd7d004229b0cb57d09b2736d8de3d8d7bdee60fde9b2 WatchSource:0}: Error finding container 8ffb960804bab04507afd7d004229b0cb57d09b2736d8de3d8d7bdee60fde9b2: Status 404 returned error can't find the container with id 8ffb960804bab04507afd7d004229b0cb57d09b2736d8de3d8d7bdee60fde9b2
Sep 10 15:07:14.376577 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.376555    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:07:14.417960 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.417913    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:14.542152 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.542122    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:14.660979 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.660948    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"]
Sep 10 15:07:14.665338 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.665321    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:14.670085 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.670062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-cb5c8c\""
Sep 10 15:07:14.670354 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.670334    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-cb5c8c-dockercfg-rkbwq\""
Sep 10 15:07:14.676136 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.676109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt" event={"ID":"71c5e681-023c-47a3-a857-4f4f51cdb7b2","Type":"ContainerStarted","Data":"8ffb960804bab04507afd7d004229b0cb57d09b2736d8de3d8d7bdee60fde9b2"}
Sep 10 15:07:14.697233 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.697203    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"]
Sep 10 15:07:14.772620 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.772584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkbb\" (UniqueName: \"kubernetes.io/projected/371dc4c1-a7e3-41d3-8197-f67b8c07a394-kube-api-access-kfkbb\") pod \"payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7\" (UID: \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:14.772802 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.772706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371dc4c1-a7e3-41d3-8197-f67b8c07a394-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7\" (UID: \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:14.873722 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.873684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkbb\" (UniqueName: \"kubernetes.io/projected/371dc4c1-a7e3-41d3-8197-f67b8c07a394-kube-api-access-kfkbb\") pod \"payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7\" (UID: \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:14.874177 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.873755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371dc4c1-a7e3-41d3-8197-f67b8c07a394-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7\" (UID: \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:14.874609 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.874586    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371dc4c1-a7e3-41d3-8197-f67b8c07a394-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7\" (UID: \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:14.888267 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.888241    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkbb\" (UniqueName: \"kubernetes.io/projected/371dc4c1-a7e3-41d3-8197-f67b8c07a394-kube-api-access-kfkbb\") pod \"payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7\" (UID: \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:14.976436 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:14.976409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:15.261041 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:15.260955    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"]
Sep 10 15:07:15.264878 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:07:15.264849    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371dc4c1_a7e3_41d3_8197_f67b8c07a394.slice/crio-512f65fd6bc9f4895cd5074348bb2861d9477bb8b81119f51ce02f2ce52c3971 WatchSource:0}: Error finding container 512f65fd6bc9f4895cd5074348bb2861d9477bb8b81119f51ce02f2ce52c3971: Status 404 returned error can't find the container with id 512f65fd6bc9f4895cd5074348bb2861d9477bb8b81119f51ce02f2ce52c3971
Sep 10 15:07:15.689603 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:15.689553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt" event={"ID":"71c5e681-023c-47a3-a857-4f4f51cdb7b2","Type":"ContainerStarted","Data":"6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364"}
Sep 10 15:07:15.689948 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:15.689680    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:15.691337 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:15.691306    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7" event={"ID":"371dc4c1-a7e3-41d3-8197-f67b8c07a394","Type":"ContainerStarted","Data":"9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c"}
Sep 10 15:07:15.691446 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:15.691352    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7" event={"ID":"371dc4c1-a7e3-41d3-8197-f67b8c07a394","Type":"ContainerStarted","Data":"512f65fd6bc9f4895cd5074348bb2861d9477bb8b81119f51ce02f2ce52c3971"}
Sep 10 15:07:15.691446 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:15.691413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:15.716019 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:15.715960    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt" podStartSLOduration=2.128416717 podStartE2EDuration="2.71594504s" totalImagesPullingTime="587.528323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:15.713755415 +0000 UTC m=+1506.190815360" watchObservedRunningTime="2026-09-10 15:07:15.71594504 +0000 UTC m=+1506.193005044"
Sep 10 15:07:15.737009 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:15.736955    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7" podStartSLOduration=1.736937834 podStartE2EDuration="1.736937834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:07:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:07:15.734437978 +0000 UTC m=+1506.211497922" watchObservedRunningTime="2026-09-10 15:07:15.736937834 +0000 UTC m=+1506.213997918"
Sep 10 15:07:21.701564 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:21.701536    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:07:26.698536 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:26.698489    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:07:37.609385 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:37.609281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:07:54.113073 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:07:54.113033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:49.220334 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:49.220302    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx_2edea59c-718b-4ce3-90a9-699f9f56aa68/payload-processing/0.log"
Sep 10 15:08:49.734446 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:49.734407    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:50.010153 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.010069    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"]
Sep 10 15:08:50.014283 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.014260    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.016646 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.016624    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-799b5a-openshift-default-dockercfg-d6nkc\""
Sep 10 15:08:50.033705 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.033682    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"]
Sep 10 15:08:50.047501 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.047477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:50.063561 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.063537    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsxt8\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-kube-api-access-zsxt8\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.063693 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.063576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-envoy\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.063693 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.063600    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-data\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.063822 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.063684    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-credential-socket\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.063822 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.063747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-certs\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.063822 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.063786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8df1d4-b609-4564-9c1b-59ad81270c15-istiod-ca-cert\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.064010 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.063819    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-podinfo\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.064010 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.063865    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-socket\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.064010 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.063959    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-token\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.164585 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.164553    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-credential-socket\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.164839 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.164603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-certs\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.164839 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.164628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8df1d4-b609-4564-9c1b-59ad81270c15-istiod-ca-cert\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.164839 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.164679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-podinfo\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.164839 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.164700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-socket\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.164839 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.164813    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-token\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.165132 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.164847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsxt8\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-kube-api-access-zsxt8\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.165132 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.164888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-envoy\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.165132 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.164919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-data\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.165132 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.165011    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-certs\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.165132 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.165080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-credential-socket\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.165403 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.165259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-data\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.165462 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.165447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8df1d4-b609-4564-9c1b-59ad81270c15-istiod-ca-cert\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.165550 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.165466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-socket\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.167095 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.167075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-envoy\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.167238 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.167221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-podinfo\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.172479 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.172455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-token\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.172611 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.172460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsxt8\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-kube-api-access-zsxt8\") pod \"e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") " pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.324199 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.324127    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:50.351140 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.351087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:50.605017 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.604989    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"]
Sep 10 15:08:50.607120 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:08:50.607092    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8df1d4_b609_4564_9c1b_59ad81270c15.slice/crio-4466c0dd004063fcd50a2586e3b8e987416b8e3663967747b8346ece67c888f7 WatchSource:0}: Error finding container 4466c0dd004063fcd50a2586e3b8e987416b8e3663967747b8346ece67c888f7: Status 404 returned error can't find the container with id 4466c0dd004063fcd50a2586e3b8e987416b8e3663967747b8346ece67c888f7
Sep 10 15:08:50.609385 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.609355    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:08:50.609478 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.609423    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:08:50.609478 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:50.609453    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:08:51.190762 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:51.190727    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" event={"ID":"3c8df1d4-b609-4564-9c1b-59ad81270c15","Type":"ContainerStarted","Data":"c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a"}
Sep 10 15:08:51.190929 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:51.190779    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" event={"ID":"3c8df1d4-b609-4564-9c1b-59ad81270c15","Type":"ContainerStarted","Data":"4466c0dd004063fcd50a2586e3b8e987416b8e3663967747b8346ece67c888f7"}
Sep 10 15:08:51.211405 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:51.211341    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" podStartSLOduration=2.211322711 podStartE2EDuration="2.211322711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:08:51.210588784 +0000 UTC m=+1601.687648727" watchObservedRunningTime="2026-09-10 15:08:51.211322711 +0000 UTC m=+1601.688382657"
Sep 10 15:08:51.324309 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:51.324276    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:51.330796 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:51.330770    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:52.196347 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:52.196305    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:52.197200 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:52.197177    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:08:53.123217 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:53.123179    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:53.215065 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:53.215028    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:54.322475 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.322439    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:54.421773 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.421742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:54.575322 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.575232    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:54.827329 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.827244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:54.934339 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.934305    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:08:54.972526 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.972485    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"]
Sep 10 15:08:54.976057 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.976041    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:08:54.979867 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.979842    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-799b5a\""
Sep 10 15:08:54.980019 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.979909    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-799b5a-dockercfg-cwhzt\""
Sep 10 15:08:54.989034 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:54.989013    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"]
Sep 10 15:08:55.012422 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:55.012382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd672e5-4573-411e-8e77-82ac6bfc7529-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts\" (UID: \"bfd672e5-4573-411e-8e77-82ac6bfc7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:08:55.012618 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:55.012441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2zvg\" (UniqueName: \"kubernetes.io/projected/bfd672e5-4573-411e-8e77-82ac6bfc7529-kube-api-access-d2zvg\") pod \"payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts\" (UID: \"bfd672e5-4573-411e-8e77-82ac6bfc7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:08:55.114030 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:55.113939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd672e5-4573-411e-8e77-82ac6bfc7529-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts\" (UID: \"bfd672e5-4573-411e-8e77-82ac6bfc7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:08:55.114030 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:55.113991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2zvg\" (UniqueName: \"kubernetes.io/projected/bfd672e5-4573-411e-8e77-82ac6bfc7529-kube-api-access-d2zvg\") pod \"payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts\" (UID: \"bfd672e5-4573-411e-8e77-82ac6bfc7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:08:55.114986 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:55.114963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd672e5-4573-411e-8e77-82ac6bfc7529-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts\" (UID: \"bfd672e5-4573-411e-8e77-82ac6bfc7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:08:55.122144 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:55.122113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2zvg\" (UniqueName: \"kubernetes.io/projected/bfd672e5-4573-411e-8e77-82ac6bfc7529-kube-api-access-d2zvg\") pod \"payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts\" (UID: \"bfd672e5-4573-411e-8e77-82ac6bfc7529\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:08:55.287441 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:55.287404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:08:55.568693 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:55.568667    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"]
Sep 10 15:08:55.570547 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:08:55.570480    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd672e5_4573_411e_8e77_82ac6bfc7529.slice/crio-0367006ffaec2c9474f56ad6f3d0e55fe243420282f9d7f22d32b32a87f14795 WatchSource:0}: Error finding container 0367006ffaec2c9474f56ad6f3d0e55fe243420282f9d7f22d32b32a87f14795: Status 404 returned error can't find the container with id 0367006ffaec2c9474f56ad6f3d0e55fe243420282f9d7f22d32b32a87f14795
Sep 10 15:08:56.220257 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:56.220227    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts" event={"ID":"bfd672e5-4573-411e-8e77-82ac6bfc7529","Type":"ContainerStarted","Data":"c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9"}
Sep 10 15:08:56.220439 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:56.220283    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts" event={"ID":"bfd672e5-4573-411e-8e77-82ac6bfc7529","Type":"ContainerStarted","Data":"0367006ffaec2c9474f56ad6f3d0e55fe243420282f9d7f22d32b32a87f14795"}
Sep 10 15:08:56.220439 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:56.220358    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:08:56.236254 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:08:56.236194    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts" podStartSLOduration=2.23617569 podStartE2EDuration="2.23617569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:08:56.235687835 +0000 UTC m=+1606.712747777" watchObservedRunningTime="2026-09-10 15:08:56.23617569 +0000 UTC m=+1606.713235635"
Sep 10 15:09:07.227772 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:07.227678    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:09:13.312006 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:13.311970    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerID="c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a" exitCode=0
Sep 10 15:09:13.312607 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:13.312037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" event={"ID":"3c8df1d4-b609-4564-9c1b-59ad81270c15","Type":"ContainerDied","Data":"c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a"}
Sep 10 15:09:13.312607 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:13.312461    2563 scope.go:117] "RemoveContainer" containerID="c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a"
Sep 10 15:09:13.313579 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:13.313553    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:09:13.313666 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:13.313614    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:09:13.313666 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:13.313644    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 15:09:14.324285 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:14.324245    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" event={"ID":"3c8df1d4-b609-4564-9c1b-59ad81270c15","Type":"ContainerStarted","Data":"ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327"}
Sep 10 15:09:14.325669 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:14.325646    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:09:14.325840 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:14.325719    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:09:14.325970 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:14.325950    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 10 15:09:15.325363 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:15.325327    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:09:15.325997 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:15.325948    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 10 15:09:16.325616 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:16.325576    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:09:16.326108 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:16.325643    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 10 15:09:17.325239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:17.325200    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:09:17.326788 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:17.326753    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 10 15:09:18.125617 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:18.125570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:18.324542 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:18.324482    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 10 15:09:18.324747 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:18.324572    2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 10 15:09:19.476378 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:19.476335    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:09:19.476938 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:19.476687    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:09:19.477735 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:19.477715    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:09:24.890734 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:24.890692    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:24.940162 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:24.940128    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"]
Sep 10 15:09:24.940413 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:24.940387    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts" podUID="bfd672e5-4573-411e-8e77-82ac6bfc7529" containerName="payload-processing" containerID="cri-o://c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9" gracePeriod=30
Sep 10 15:09:24.968788 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:24.968752    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:25.106485 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.106428    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:25.214369 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.214347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:09:25.228776 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.228744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:25.323385 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.323354    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2zvg\" (UniqueName: \"kubernetes.io/projected/bfd672e5-4573-411e-8e77-82ac6bfc7529-kube-api-access-d2zvg\") pod \"bfd672e5-4573-411e-8e77-82ac6bfc7529\" (UID: \"bfd672e5-4573-411e-8e77-82ac6bfc7529\") "
Sep 10 15:09:25.323615 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.323442    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd672e5-4573-411e-8e77-82ac6bfc7529-plugins-config-volume\") pod \"bfd672e5-4573-411e-8e77-82ac6bfc7529\" (UID: \"bfd672e5-4573-411e-8e77-82ac6bfc7529\") "
Sep 10 15:09:25.323874 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.323851    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd672e5-4573-411e-8e77-82ac6bfc7529-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfd672e5-4573-411e-8e77-82ac6bfc7529" (UID: "bfd672e5-4573-411e-8e77-82ac6bfc7529"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:09:25.325686 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.325658    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd672e5-4573-411e-8e77-82ac6bfc7529-kube-api-access-d2zvg" (OuterVolumeSpecName: "kube-api-access-d2zvg") pod "bfd672e5-4573-411e-8e77-82ac6bfc7529" (UID: "bfd672e5-4573-411e-8e77-82ac6bfc7529"). InnerVolumeSpecName "kube-api-access-d2zvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:09:25.388425 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.388389    2563 generic.go:356] "Generic (PLEG): container finished" podID="bfd672e5-4573-411e-8e77-82ac6bfc7529" containerID="c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9" exitCode=0
Sep 10 15:09:25.388613 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.388439    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"
Sep 10 15:09:25.388613 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.388456    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts" event={"ID":"bfd672e5-4573-411e-8e77-82ac6bfc7529","Type":"ContainerDied","Data":"c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9"}
Sep 10 15:09:25.388613 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.388532    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts" event={"ID":"bfd672e5-4573-411e-8e77-82ac6bfc7529","Type":"ContainerDied","Data":"0367006ffaec2c9474f56ad6f3d0e55fe243420282f9d7f22d32b32a87f14795"}
Sep 10 15:09:25.388613 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.388564    2563 scope.go:117] "RemoveContainer" containerID="c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9"
Sep 10 15:09:25.398220 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.398198    2563 scope.go:117] "RemoveContainer" containerID="c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9"
Sep 10 15:09:25.398479 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:09:25.398458    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9\": container with ID starting with c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9 not found: ID does not exist" containerID="c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9"
Sep 10 15:09:25.398559 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.398488    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9"} err="failed to get container status \"c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9\": rpc error: code = NotFound desc = could not find container \"c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9\": container with ID starting with c8df69d8d2bc205af575b2ddae449b239f9533f1d8c854dcd5feee56e169e8c9 not found: ID does not exist"
Sep 10 15:09:25.411071 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.411046    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"]
Sep 10 15:09:25.417319 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.417292    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-799b5a-7695648cbf-mw9ts"]
Sep 10 15:09:25.425254 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.425230    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2zvg\" (UniqueName: \"kubernetes.io/projected/bfd672e5-4573-411e-8e77-82ac6bfc7529-kube-api-access-d2zvg\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:25.425347 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.425255    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfd672e5-4573-411e-8e77-82ac6bfc7529-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:25.574538 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:25.574448    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:26.033093 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:26.033063    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfd672e5-4573-411e-8e77-82ac6bfc7529" path="/var/lib/kubelet/pods/bfd672e5-4573-411e-8e77-82ac6bfc7529/volumes"
Sep 10 15:09:31.688141 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:31.688104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:31.757128 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:31.757094    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:31.945111 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:31.945034    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"]
Sep 10 15:09:31.945355 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:31.945328    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy" containerID="cri-o://ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327" gracePeriod=30
Sep 10 15:09:31.967098 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:31.967057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:34.478628 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:34.478591    2563 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:09:34.478998 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:34.478656    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:09:35.837482 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:35.837452    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:37.206952 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.206887    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:09:37.335138 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335101    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-podinfo\") pod \"3c8df1d4-b609-4564-9c1b-59ad81270c15\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") "
Sep 10 15:09:37.335324 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335159    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-token\") pod \"3c8df1d4-b609-4564-9c1b-59ad81270c15\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") "
Sep 10 15:09:37.335324 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335202    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-socket\") pod \"3c8df1d4-b609-4564-9c1b-59ad81270c15\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") "
Sep 10 15:09:37.335324 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335220    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-data\") pod \"3c8df1d4-b609-4564-9c1b-59ad81270c15\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") "
Sep 10 15:09:37.335324 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsxt8\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-kube-api-access-zsxt8\") pod \"3c8df1d4-b609-4564-9c1b-59ad81270c15\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") "
Sep 10 15:09:37.335324 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-envoy\") pod \"3c8df1d4-b609-4564-9c1b-59ad81270c15\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") "
Sep 10 15:09:37.335601 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335368    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-credential-socket\") pod \"3c8df1d4-b609-4564-9c1b-59ad81270c15\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") "
Sep 10 15:09:37.335601 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335403    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-certs\") pod \"3c8df1d4-b609-4564-9c1b-59ad81270c15\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") "
Sep 10 15:09:37.335601 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335453    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8df1d4-b609-4564-9c1b-59ad81270c15-istiod-ca-cert\") pod \"3c8df1d4-b609-4564-9c1b-59ad81270c15\" (UID: \"3c8df1d4-b609-4564-9c1b-59ad81270c15\") "
Sep 10 15:09:37.335601 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335586    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3c8df1d4-b609-4564-9c1b-59ad81270c15" (UID: "3c8df1d4-b609-4564-9c1b-59ad81270c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:09:37.335811 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-data" (OuterVolumeSpecName: "istio-data") pod "3c8df1d4-b609-4564-9c1b-59ad81270c15" (UID: "3c8df1d4-b609-4564-9c1b-59ad81270c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:09:37.335892 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335872    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:37.335957 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335892    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-data\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:37.336015 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335956    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8df1d4-b609-4564-9c1b-59ad81270c15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3c8df1d4-b609-4564-9c1b-59ad81270c15" (UID: "3c8df1d4-b609-4564-9c1b-59ad81270c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:09:37.336015 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.335987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3c8df1d4-b609-4564-9c1b-59ad81270c15" (UID: "3c8df1d4-b609-4564-9c1b-59ad81270c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:09:37.336127 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.336102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3c8df1d4-b609-4564-9c1b-59ad81270c15" (UID: "3c8df1d4-b609-4564-9c1b-59ad81270c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:09:37.337671 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.337641    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3c8df1d4-b609-4564-9c1b-59ad81270c15" (UID: "3c8df1d4-b609-4564-9c1b-59ad81270c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:09:37.337804 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.337649    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-token" (OuterVolumeSpecName: "istio-token") pod "3c8df1d4-b609-4564-9c1b-59ad81270c15" (UID: "3c8df1d4-b609-4564-9c1b-59ad81270c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:09:37.337876 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.337857    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3c8df1d4-b609-4564-9c1b-59ad81270c15" (UID: "3c8df1d4-b609-4564-9c1b-59ad81270c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:09:37.337990 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.337970    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-kube-api-access-zsxt8" (OuterVolumeSpecName: "kube-api-access-zsxt8") pod "3c8df1d4-b609-4564-9c1b-59ad81270c15" (UID: "3c8df1d4-b609-4564-9c1b-59ad81270c15"). InnerVolumeSpecName "kube-api-access-zsxt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:09:37.437363 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.437327    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-podinfo\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:37.437363 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.437358    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-token\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:37.437363 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.437366    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsxt8\" (UniqueName: \"kubernetes.io/projected/3c8df1d4-b609-4564-9c1b-59ad81270c15-kube-api-access-zsxt8\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:37.437657 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.437376    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-istio-envoy\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:37.437657 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.437384    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-credential-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:37.437657 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.437392    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8df1d4-b609-4564-9c1b-59ad81270c15-workload-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:37.437657 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.437399    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8df1d4-b609-4564-9c1b-59ad81270c15-istiod-ca-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:37.453880 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.453847    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerID="ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327" exitCode=0
Sep 10 15:09:37.454044 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.453898    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"
Sep 10 15:09:37.454044 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.453935    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" event={"ID":"3c8df1d4-b609-4564-9c1b-59ad81270c15","Type":"ContainerDied","Data":"ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327"}
Sep 10 15:09:37.454044 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.453993    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn" event={"ID":"3c8df1d4-b609-4564-9c1b-59ad81270c15","Type":"ContainerDied","Data":"4466c0dd004063fcd50a2586e3b8e987416b8e3663967747b8346ece67c888f7"}
Sep 10 15:09:37.454044 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.454024    2563 scope.go:117] "RemoveContainer" containerID="ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327"
Sep 10 15:09:37.463674 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.463646    2563 scope.go:117] "RemoveContainer" containerID="c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a"
Sep 10 15:09:37.472197 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.472179    2563 scope.go:117] "RemoveContainer" containerID="ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327"
Sep 10 15:09:37.472469 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:09:37.472451    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327\": container with ID starting with ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327 not found: ID does not exist" containerID="ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327"
Sep 10 15:09:37.472546 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.472479    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327"} err="failed to get container status \"ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327\": rpc error: code = NotFound desc = could not find container \"ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327\": container with ID starting with ba1a1853b5f6000c3cde016f8ffd8cdcfae7fbe347346faed8f15ec82b01e327 not found: ID does not exist"
Sep 10 15:09:37.472546 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.472499    2563 scope.go:117] "RemoveContainer" containerID="c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a"
Sep 10 15:09:37.472893 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:09:37.472823    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a\": container with ID starting with c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a not found: ID does not exist" containerID="c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a"
Sep 10 15:09:37.472893 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.472855    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a"} err="failed to get container status \"c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a\": rpc error: code = NotFound desc = could not find container \"c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a\": container with ID starting with c570d79b3b32da520bd5d5d59f721f008652bdb506a61376ed7b91cde1dc235a not found: ID does not exist"
Sep 10 15:09:37.475386 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.475364    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"]
Sep 10 15:09:37.479688 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:37.479668    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-799b5a-openshift-default-77cf5c6b58-plsmn"]
Sep 10 15:09:38.032715 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:38.032685    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" path="/var/lib/kubelet/pods/3c8df1d4-b609-4564-9c1b-59ad81270c15/volumes"
Sep 10 15:09:51.290402 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.290365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:51.314762 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.314728    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"]
Sep 10 15:09:51.315010 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.314983    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx" podUID="2edea59c-718b-4ce3-90a9-699f9f56aa68" containerName="payload-processing" containerID="cri-o://510bb0570f29f852d40d42081759ac237e2897bb8efa8623062cc453a91412b2" gracePeriod=30
Sep 10 15:09:51.322603 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.322575    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"]
Sep 10 15:09:51.323033 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.323004    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z" podUID="7b612830-9f63-4cb5-8b50-19d06ffa789f" containerName="payload-pre-processing" containerID="cri-o://405dffbd7282df0630e76a31275192d62ff3d8d9d76a8b2ac5900bf209f3e359" gracePeriod=30
Sep 10 15:09:51.391497 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.391445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:51.491778 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.491698    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:51.536293 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.536266    2563 generic.go:356] "Generic (PLEG): container finished" podID="7b612830-9f63-4cb5-8b50-19d06ffa789f" containerID="405dffbd7282df0630e76a31275192d62ff3d8d9d76a8b2ac5900bf209f3e359" exitCode=0
Sep 10 15:09:51.536408 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.536336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z" event={"ID":"7b612830-9f63-4cb5-8b50-19d06ffa789f","Type":"ContainerDied","Data":"405dffbd7282df0630e76a31275192d62ff3d8d9d76a8b2ac5900bf209f3e359"}
Sep 10 15:09:51.538774 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.538751    2563 generic.go:356] "Generic (PLEG): container finished" podID="2edea59c-718b-4ce3-90a9-699f9f56aa68" containerID="510bb0570f29f852d40d42081759ac237e2897bb8efa8623062cc453a91412b2" exitCode=0
Sep 10 15:09:51.538905 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.538863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx" event={"ID":"2edea59c-718b-4ce3-90a9-699f9f56aa68","Type":"ContainerDied","Data":"510bb0570f29f852d40d42081759ac237e2897bb8efa8623062cc453a91412b2"}
Sep 10 15:09:51.622260 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.622232    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:09:51.623312 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.623296    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:09:51.624460 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.624436    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:51.769610 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.769497    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b612830-9f63-4cb5-8b50-19d06ffa789f-plugins-config-volume\") pod \"7b612830-9f63-4cb5-8b50-19d06ffa789f\" (UID: \"7b612830-9f63-4cb5-8b50-19d06ffa789f\") "
Sep 10 15:09:51.769784 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.769634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8tcv\" (UniqueName: \"kubernetes.io/projected/7b612830-9f63-4cb5-8b50-19d06ffa789f-kube-api-access-f8tcv\") pod \"7b612830-9f63-4cb5-8b50-19d06ffa789f\" (UID: \"7b612830-9f63-4cb5-8b50-19d06ffa789f\") "
Sep 10 15:09:51.769784 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.769673    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgmkb\" (UniqueName: \"kubernetes.io/projected/2edea59c-718b-4ce3-90a9-699f9f56aa68-kube-api-access-jgmkb\") pod \"2edea59c-718b-4ce3-90a9-699f9f56aa68\" (UID: \"2edea59c-718b-4ce3-90a9-699f9f56aa68\") "
Sep 10 15:09:51.769784 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.769691    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edea59c-718b-4ce3-90a9-699f9f56aa68-plugins-config-volume\") pod \"2edea59c-718b-4ce3-90a9-699f9f56aa68\" (UID: \"2edea59c-718b-4ce3-90a9-699f9f56aa68\") "
Sep 10 15:09:51.769943 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.769904    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b612830-9f63-4cb5-8b50-19d06ffa789f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b612830-9f63-4cb5-8b50-19d06ffa789f" (UID: "7b612830-9f63-4cb5-8b50-19d06ffa789f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:09:51.770095 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.770072    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2edea59c-718b-4ce3-90a9-699f9f56aa68-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2edea59c-718b-4ce3-90a9-699f9f56aa68" (UID: "2edea59c-718b-4ce3-90a9-699f9f56aa68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:09:51.771972 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.771940    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b612830-9f63-4cb5-8b50-19d06ffa789f-kube-api-access-f8tcv" (OuterVolumeSpecName: "kube-api-access-f8tcv") pod "7b612830-9f63-4cb5-8b50-19d06ffa789f" (UID: "7b612830-9f63-4cb5-8b50-19d06ffa789f"). InnerVolumeSpecName "kube-api-access-f8tcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:09:51.772078 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.771976    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edea59c-718b-4ce3-90a9-699f9f56aa68-kube-api-access-jgmkb" (OuterVolumeSpecName: "kube-api-access-jgmkb") pod "2edea59c-718b-4ce3-90a9-699f9f56aa68" (UID: "2edea59c-718b-4ce3-90a9-699f9f56aa68"). InnerVolumeSpecName "kube-api-access-jgmkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:09:51.862777 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.862745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:09:51.870819 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.870792    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b612830-9f63-4cb5-8b50-19d06ffa789f-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:51.870819 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.870819    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8tcv\" (UniqueName: \"kubernetes.io/projected/7b612830-9f63-4cb5-8b50-19d06ffa789f-kube-api-access-f8tcv\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:51.870985 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.870832    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgmkb\" (UniqueName: \"kubernetes.io/projected/2edea59c-718b-4ce3-90a9-699f9f56aa68-kube-api-access-jgmkb\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:51.870985 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:51.870843    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2edea59c-718b-4ce3-90a9-699f9f56aa68-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:09:52.545347 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.545320    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"
Sep 10 15:09:52.545824 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.545363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx" event={"ID":"2edea59c-718b-4ce3-90a9-699f9f56aa68","Type":"ContainerDied","Data":"65c1ec3f45c4136ab8ddf829e1738b5bbab75ecaac0c4e266bad9cc93a7b4a13"}
Sep 10 15:09:52.545824 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.545432    2563 scope.go:117] "RemoveContainer" containerID="510bb0570f29f852d40d42081759ac237e2897bb8efa8623062cc453a91412b2"
Sep 10 15:09:52.554178 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.554154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z" event={"ID":"7b612830-9f63-4cb5-8b50-19d06ffa789f","Type":"ContainerDied","Data":"b6aa8c18c821be72070f97b3e10367cd3d39f64273bae120679044b0f361a159"}
Sep 10 15:09:52.554314 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.554181    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"
Sep 10 15:09:52.561180 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.561159    2563 scope.go:117] "RemoveContainer" containerID="405dffbd7282df0630e76a31275192d62ff3d8d9d76a8b2ac5900bf209f3e359"
Sep 10 15:09:52.565610 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.565585    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"]
Sep 10 15:09:52.573342 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.573323    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b022ee-5bf9c484fc-6v6jx"]
Sep 10 15:09:52.583519 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.583487    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"]
Sep 10 15:09:52.590755 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:52.590736    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b022ee-5d8947fffd-wtr8z"]
Sep 10 15:09:54.032397 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:54.032364    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2edea59c-718b-4ce3-90a9-699f9f56aa68" path="/var/lib/kubelet/pods/2edea59c-718b-4ce3-90a9-699f9f56aa68/volumes"
Sep 10 15:09:54.032817 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:54.032783    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b612830-9f63-4cb5-8b50-19d06ffa789f" path="/var/lib/kubelet/pods/7b612830-9f63-4cb5-8b50-19d06ffa789f/volumes"
Sep 10 15:09:56.892740 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:09:56.892704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:00.867186 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:00.867146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:13.905742 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:13.905707    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:30.187055 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:30.187019    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:33.895424 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:33.895339    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:34.504867 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:34.504814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:40.158727 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:40.158674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:40.259199 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:40.259166    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:40.443212 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:40.443132    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"]
Sep 10 15:10:40.443555 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:40.443499    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy" containerID="cri-o://b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a" gracePeriod=30
Sep 10 15:10:40.506838 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:40.506797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:40.903266 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:40.903226    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:41.787318 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:41.787273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:45.697008 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.696986    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:10:45.835156 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.835060    2563 generic.go:356] "Generic (PLEG): container finished" podID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerID="b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a" exitCode=0
Sep 10 15:10:45.835156 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.835111    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"
Sep 10 15:10:45.835156 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.835119    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" event={"ID":"998f4479-b372-4c83-b1da-e6c71bc16d70","Type":"ContainerDied","Data":"b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a"}
Sep 10 15:10:45.835448 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.835162    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4" event={"ID":"998f4479-b372-4c83-b1da-e6c71bc16d70","Type":"ContainerDied","Data":"4f56055ca7e3a9912a8ea7480e30f9ceaca2f3b9a55467441fc1d449e599fb1f"}
Sep 10 15:10:45.835448 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.835183    2563 scope.go:117] "RemoveContainer" containerID="b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a"
Sep 10 15:10:45.843823 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.843805    2563 scope.go:117] "RemoveContainer" containerID="8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea"
Sep 10 15:10:45.851761 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.851743    2563 scope.go:117] "RemoveContainer" containerID="b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a"
Sep 10 15:10:45.852031 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:10:45.852014    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a\": container with ID starting with b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a not found: ID does not exist" containerID="b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a"
Sep 10 15:10:45.852077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.852048    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a"} err="failed to get container status \"b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a\": rpc error: code = NotFound desc = could not find container \"b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a\": container with ID starting with b573e20bb7779908a7d99761601bd2c17bc88e18d402c9ff1446f314a542043a not found: ID does not exist"
Sep 10 15:10:45.852077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.852065    2563 scope.go:117] "RemoveContainer" containerID="8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea"
Sep 10 15:10:45.852290 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:10:45.852271    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea\": container with ID starting with 8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea not found: ID does not exist" containerID="8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea"
Sep 10 15:10:45.852327 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.852298    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea"} err="failed to get container status \"8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea\": rpc error: code = NotFound desc = could not find container \"8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea\": container with ID starting with 8a5c7589911fde1e6e17a53f8b8b7418f9b261396aff5880e3f491ae0cdc01ea not found: ID does not exist"
Sep 10 15:10:45.868679 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.868659    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-socket\") pod \"998f4479-b372-4c83-b1da-e6c71bc16d70\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") "
Sep 10 15:10:45.868758 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.868715    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-data\") pod \"998f4479-b372-4c83-b1da-e6c71bc16d70\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") "
Sep 10 15:10:45.868758 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.868747    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-kube-api-access-dmd6k\") pod \"998f4479-b372-4c83-b1da-e6c71bc16d70\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") "
Sep 10 15:10:45.868830 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.868790    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-certs\") pod \"998f4479-b372-4c83-b1da-e6c71bc16d70\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") "
Sep 10 15:10:45.868874 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.868859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-podinfo\") pod \"998f4479-b372-4c83-b1da-e6c71bc16d70\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") "
Sep 10 15:10:45.868921 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.868889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-credential-socket\") pod \"998f4479-b372-4c83-b1da-e6c71bc16d70\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") "
Sep 10 15:10:45.868921 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.868914    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-envoy\") pod \"998f4479-b372-4c83-b1da-e6c71bc16d70\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") "
Sep 10 15:10:45.869024 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.868935    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "998f4479-b372-4c83-b1da-e6c71bc16d70" (UID: "998f4479-b372-4c83-b1da-e6c71bc16d70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:45.869024 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.868939    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-token\") pod \"998f4479-b372-4c83-b1da-e6c71bc16d70\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") "
Sep 10 15:10:45.869024 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.869005    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998f4479-b372-4c83-b1da-e6c71bc16d70-istiod-ca-cert\") pod \"998f4479-b372-4c83-b1da-e6c71bc16d70\" (UID: \"998f4479-b372-4c83-b1da-e6c71bc16d70\") "
Sep 10 15:10:45.869181 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.869017    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-data" (OuterVolumeSpecName: "istio-data") pod "998f4479-b372-4c83-b1da-e6c71bc16d70" (UID: "998f4479-b372-4c83-b1da-e6c71bc16d70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:45.869181 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.869113    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "998f4479-b372-4c83-b1da-e6c71bc16d70" (UID: "998f4479-b372-4c83-b1da-e6c71bc16d70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:45.869295 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.869229    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "998f4479-b372-4c83-b1da-e6c71bc16d70" (UID: "998f4479-b372-4c83-b1da-e6c71bc16d70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:45.869420 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.869393    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998f4479-b372-4c83-b1da-e6c71bc16d70-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "998f4479-b372-4c83-b1da-e6c71bc16d70" (UID: "998f4479-b372-4c83-b1da-e6c71bc16d70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:10:45.869548 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.869476    2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-data\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:45.869548 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.869495    2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:45.869548 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.869536    2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-credential-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:45.869697 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.869551    2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-workload-socket\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:45.870907 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.870886    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-kube-api-access-dmd6k" (OuterVolumeSpecName: "kube-api-access-dmd6k") pod "998f4479-b372-4c83-b1da-e6c71bc16d70" (UID: "998f4479-b372-4c83-b1da-e6c71bc16d70"). InnerVolumeSpecName "kube-api-access-dmd6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:10:45.871075 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.871048    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "998f4479-b372-4c83-b1da-e6c71bc16d70" (UID: "998f4479-b372-4c83-b1da-e6c71bc16d70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 15:10:45.871356 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.871340    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-token" (OuterVolumeSpecName: "istio-token") pod "998f4479-b372-4c83-b1da-e6c71bc16d70" (UID: "998f4479-b372-4c83-b1da-e6c71bc16d70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:10:45.871423 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.871361    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "998f4479-b372-4c83-b1da-e6c71bc16d70" (UID: "998f4479-b372-4c83-b1da-e6c71bc16d70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:10:45.970466 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.970432    2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-podinfo\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:45.970466 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.970459    2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-envoy\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:45.970466 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.970470    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-istio-token\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:45.970466 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.970477    2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/998f4479-b372-4c83-b1da-e6c71bc16d70-istiod-ca-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:45.970740 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:45.970485    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmd6k\" (UniqueName: \"kubernetes.io/projected/998f4479-b372-4c83-b1da-e6c71bc16d70-kube-api-access-dmd6k\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:46.152032 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:46.152000    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"]
Sep 10 15:10:46.157673 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:46.157648    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b022ee-openshift-default-7966dcf58-lnww4"]
Sep 10 15:10:48.032104 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:48.032068    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" path="/var/lib/kubelet/pods/998f4479-b372-4c83-b1da-e6c71bc16d70/volumes"
Sep 10 15:10:51.282001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.281367    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"]
Sep 10 15:10:51.282001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.281638    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt" podUID="71c5e681-023c-47a3-a857-4f4f51cdb7b2" containerName="maas-api" containerID="cri-o://6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364" gracePeriod=30
Sep 10 15:10:51.315386 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.315352    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:51.352598 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.352569    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"]
Sep 10 15:10:51.352837 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.352790    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7" podUID="371dc4c1-a7e3-41d3-8197-f67b8c07a394" containerName="payload-pre-processing" containerID="cri-o://9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c" gracePeriod=30
Sep 10 15:10:51.391845 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.391810    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:51.495995 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.495959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:51.576651 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.576476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:51.617913 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.617887    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:10:51.620997 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.620977    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6p7p\" (UniqueName: \"kubernetes.io/projected/71c5e681-023c-47a3-a857-4f4f51cdb7b2-kube-api-access-j6p7p\") pod \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") "
Sep 10 15:10:51.621077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.621032    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-maas-api-tls\") pod \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") "
Sep 10 15:10:51.621077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.621058    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-metrics-certs\") pod \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") "
Sep 10 15:10:51.621174 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.621158    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c5e681-023c-47a3-a857-4f4f51cdb7b2-openshift-service-ca\") pod \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\" (UID: \"71c5e681-023c-47a3-a857-4f4f51cdb7b2\") "
Sep 10 15:10:51.621662 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.621637    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71c5e681-023c-47a3-a857-4f4f51cdb7b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "71c5e681-023c-47a3-a857-4f4f51cdb7b2" (UID: "71c5e681-023c-47a3-a857-4f4f51cdb7b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:10:51.623640 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.623615    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "71c5e681-023c-47a3-a857-4f4f51cdb7b2" (UID: "71c5e681-023c-47a3-a857-4f4f51cdb7b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:10:51.624326 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.624302    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71c5e681-023c-47a3-a857-4f4f51cdb7b2-kube-api-access-j6p7p" (OuterVolumeSpecName: "kube-api-access-j6p7p") pod "71c5e681-023c-47a3-a857-4f4f51cdb7b2" (UID: "71c5e681-023c-47a3-a857-4f4f51cdb7b2"). InnerVolumeSpecName "kube-api-access-j6p7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:10:51.624721 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.624689    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "71c5e681-023c-47a3-a857-4f4f51cdb7b2" (UID: "71c5e681-023c-47a3-a857-4f4f51cdb7b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:10:51.640086 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.639734    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:10:51.722370 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.722288    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfkbb\" (UniqueName: \"kubernetes.io/projected/371dc4c1-a7e3-41d3-8197-f67b8c07a394-kube-api-access-kfkbb\") pod \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\" (UID: \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\") "
Sep 10 15:10:51.722560 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.722544    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:51.722616 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.722564    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/71c5e681-023c-47a3-a857-4f4f51cdb7b2-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:51.722616 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.722578    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/71c5e681-023c-47a3-a857-4f4f51cdb7b2-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:51.722616 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.722592    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6p7p\" (UniqueName: \"kubernetes.io/projected/71c5e681-023c-47a3-a857-4f4f51cdb7b2-kube-api-access-j6p7p\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:51.724293 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.724268    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371dc4c1-a7e3-41d3-8197-f67b8c07a394-kube-api-access-kfkbb" (OuterVolumeSpecName: "kube-api-access-kfkbb") pod "371dc4c1-a7e3-41d3-8197-f67b8c07a394" (UID: "371dc4c1-a7e3-41d3-8197-f67b8c07a394"). InnerVolumeSpecName "kube-api-access-kfkbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:10:51.822996 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.822961    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371dc4c1-a7e3-41d3-8197-f67b8c07a394-plugins-config-volume\") pod \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\" (UID: \"371dc4c1-a7e3-41d3-8197-f67b8c07a394\") "
Sep 10 15:10:51.823198 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.823184    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfkbb\" (UniqueName: \"kubernetes.io/projected/371dc4c1-a7e3-41d3-8197-f67b8c07a394-kube-api-access-kfkbb\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:51.823294 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.823273    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371dc4c1-a7e3-41d3-8197-f67b8c07a394-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "371dc4c1-a7e3-41d3-8197-f67b8c07a394" (UID: "371dc4c1-a7e3-41d3-8197-f67b8c07a394"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:10:51.872784 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.872757    2563 generic.go:356] "Generic (PLEG): container finished" podID="71c5e681-023c-47a3-a857-4f4f51cdb7b2" containerID="6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364" exitCode=0
Sep 10 15:10:51.872953 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.872809    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"
Sep 10 15:10:51.872953 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.872851    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt" event={"ID":"71c5e681-023c-47a3-a857-4f4f51cdb7b2","Type":"ContainerDied","Data":"6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364"}
Sep 10 15:10:51.872953 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.872905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt" event={"ID":"71c5e681-023c-47a3-a857-4f4f51cdb7b2","Type":"ContainerDied","Data":"8ffb960804bab04507afd7d004229b0cb57d09b2736d8de3d8d7bdee60fde9b2"}
Sep 10 15:10:51.872953 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.872936    2563 scope.go:117] "RemoveContainer" containerID="6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364"
Sep 10 15:10:51.874369 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.874348    2563 generic.go:356] "Generic (PLEG): container finished" podID="371dc4c1-a7e3-41d3-8197-f67b8c07a394" containerID="9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c" exitCode=0
Sep 10 15:10:51.874462 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.874389    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"
Sep 10 15:10:51.874462 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.874397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7" event={"ID":"371dc4c1-a7e3-41d3-8197-f67b8c07a394","Type":"ContainerDied","Data":"9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c"}
Sep 10 15:10:51.874462 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.874430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7" event={"ID":"371dc4c1-a7e3-41d3-8197-f67b8c07a394","Type":"ContainerDied","Data":"512f65fd6bc9f4895cd5074348bb2861d9477bb8b81119f51ce02f2ce52c3971"}
Sep 10 15:10:51.883328 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.883307    2563 scope.go:117] "RemoveContainer" containerID="6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364"
Sep 10 15:10:51.883711 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:10:51.883619    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364\": container with ID starting with 6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364 not found: ID does not exist" containerID="6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364"
Sep 10 15:10:51.883711 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.883655    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364"} err="failed to get container status \"6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364\": rpc error: code = NotFound desc = could not find container \"6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364\": container with ID starting with 6c2b3d7d7b646d04a0562572d36bd321cff3620bb66125c5a421fd62b099f364 not found: ID does not exist"
Sep 10 15:10:51.883711 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.883678    2563 scope.go:117] "RemoveContainer" containerID="9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c"
Sep 10 15:10:51.891477 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.891461    2563 scope.go:117] "RemoveContainer" containerID="9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c"
Sep 10 15:10:51.891724 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:10:51.891707    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c\": container with ID starting with 9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c not found: ID does not exist" containerID="9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c"
Sep 10 15:10:51.891781 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.891729    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c"} err="failed to get container status \"9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c\": rpc error: code = NotFound desc = could not find container \"9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c\": container with ID starting with 9d4c4c1f2bc6debd2db6815cc22d64930bf8b634daf4be6587940b6d5104ae5c not found: ID does not exist"
Sep 10 15:10:51.896098 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.896077    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"]
Sep 10 15:10:51.908998 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.905695    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cb5c8c-78b978f99c-k8nkt"]
Sep 10 15:10:51.920124 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.920099    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"]
Sep 10 15:10:51.923619 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.923600    2563 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371dc4c1-a7e3-41d3-8197-f67b8c07a394-plugins-config-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:10:51.923889 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.923872    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-cb5c8c-7c9bd6984b-5ttp7"]
Sep 10 15:10:51.937873 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:51.937848    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:52.032352 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:52.032282    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="371dc4c1-a7e3-41d3-8197-f67b8c07a394" path="/var/lib/kubelet/pods/371dc4c1-a7e3-41d3-8197-f67b8c07a394/volumes"
Sep 10 15:10:52.032708 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:52.032694    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71c5e681-023c-47a3-a857-4f4f51cdb7b2" path="/var/lib/kubelet/pods/71c5e681-023c-47a3-a857-4f4f51cdb7b2/volumes"
Sep 10 15:10:57.788148 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:57.788116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:57.889179 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:57.889143    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:58.052988 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:58.052889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:10:58.579243 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:10:58.579204    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:03.107437 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:03.107401    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:08.961481 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:08.961447    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"]
Sep 10 15:11:08.962213 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:08.962170    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276" podUID="5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" containerName="maas-api" containerID="cri-o://7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d" gracePeriod=30
Sep 10 15:11:08.997277 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:08.997244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:09.076110 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.076077    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:09.147789 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.147757    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:09.222768 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.222710    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 15:11:09.227114 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.227079    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:09.247698 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.247670    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-metrics-certs\") pod \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") "
Sep 10 15:11:09.247820 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.247734    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-maas-api-tls\") pod \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") "
Sep 10 15:11:09.247820 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.247766    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vjhr\" (UniqueName: \"kubernetes.io/projected/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-kube-api-access-5vjhr\") pod \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") "
Sep 10 15:11:09.247820 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.247804    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-openshift-service-ca\") pod \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\" (UID: \"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9\") "
Sep 10 15:11:09.248277 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.248247    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" (UID: "5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:11:09.249913 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.249882    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" (UID: "5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:11:09.249996 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.249958    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" (UID: "5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:11:09.249996 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.249967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-kube-api-access-5vjhr" (OuterVolumeSpecName: "kube-api-access-5vjhr") pod "5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" (UID: "5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9"). InnerVolumeSpecName "kube-api-access-5vjhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:11:09.290683 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.290649    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:09.348595 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.348563    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:09.348595 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.348593    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vjhr\" (UniqueName: \"kubernetes.io/projected/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-kube-api-access-5vjhr\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:09.348848 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.348604    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:09.348848 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.348613    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:09.604174 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.604090    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:09.968318 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.968287    2563 generic.go:356] "Generic (PLEG): container finished" podID="5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" containerID="7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d" exitCode=0
Sep 10 15:11:09.968760 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.968328    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"
Sep 10 15:11:09.968760 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.968362    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276" event={"ID":"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9","Type":"ContainerDied","Data":"7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d"}
Sep 10 15:11:09.968760 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.968405    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276" event={"ID":"5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9","Type":"ContainerDied","Data":"08bade2f78aa299eb9cf8e5452a508516a8bc5d716c9d7c450d3159b54e36257"}
Sep 10 15:11:09.968760 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.968426    2563 scope.go:117] "RemoveContainer" containerID="7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d"
Sep 10 15:11:09.977781 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.977765    2563 scope.go:117] "RemoveContainer" containerID="7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d"
Sep 10 15:11:09.978028 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:11:09.978011    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d\": container with ID starting with 7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d not found: ID does not exist" containerID="7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d"
Sep 10 15:11:09.978064 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.978037    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d"} err="failed to get container status \"7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d\": rpc error: code = NotFound desc = could not find container \"7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d\": container with ID starting with 7da6db0858b8e1ae3979b9a02861afd681375ff8746ed21ba93fdadc0197bc0d not found: ID does not exist"
Sep 10 15:11:09.993273 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.993252    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"]
Sep 10 15:11:09.997126 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:09.997105    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-16ad45-5979559ff8-x9276"]
Sep 10 15:11:10.032058 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:10.032029    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" path="/var/lib/kubelet/pods/5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9/volumes"
Sep 10 15:11:13.662311 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:13.662270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:13.763773 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:13.763741    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:15.654520 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:15.654467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:15.720929 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:15.720877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:15.935497 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:15.935402    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:16.382681 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:16.382586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:26.804301 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:26.804263    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"]
Sep 10 15:11:26.804801 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:26.804475    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7" podUID="b78d2fb4-8068-48fc-87b7-32b6a0086f79" containerName="maas-api" containerID="cri-o://862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06" gracePeriod=30
Sep 10 15:11:26.830855 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:26.830826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:26.958708 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:26.958672    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:27.058233 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.058174    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:11:27.062918 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.062894    2563 generic.go:356] "Generic (PLEG): container finished" podID="b78d2fb4-8068-48fc-87b7-32b6a0086f79" containerID="862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06" exitCode=0
Sep 10 15:11:27.063058 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.062934    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"
Sep 10 15:11:27.063058 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.062977    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7" event={"ID":"b78d2fb4-8068-48fc-87b7-32b6a0086f79","Type":"ContainerDied","Data":"862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06"}
Sep 10 15:11:27.063058 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.063037    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7" event={"ID":"b78d2fb4-8068-48fc-87b7-32b6a0086f79","Type":"ContainerDied","Data":"85d25409a7cfe181da3d209d0cd4bd9a78d670a8372e83a1acdfd59b9cff9e6d"}
Sep 10 15:11:27.063180 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.063064    2563 scope.go:117] "RemoveContainer" containerID="862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06"
Sep 10 15:11:27.071784 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.071765    2563 scope.go:117] "RemoveContainer" containerID="862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06"
Sep 10 15:11:27.072146 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:11:27.072061    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06\": container with ID starting with 862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06 not found: ID does not exist" containerID="862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06"
Sep 10 15:11:27.072146 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.072098    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06"} err="failed to get container status \"862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06\": rpc error: code = NotFound desc = could not find container \"862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06\": container with ID starting with 862810264e8581e8920feaf526f591d65586840ba9e580120eba76e4f3cfdf06 not found: ID does not exist"
Sep 10 15:11:27.072146 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.072140    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:27.100776 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.100745    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-maas-api-tls\") pod \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") "
Sep 10 15:11:27.100928 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.100815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78d2fb4-8068-48fc-87b7-32b6a0086f79-openshift-service-ca\") pod \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") "
Sep 10 15:11:27.100928 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.100850    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdzfx\" (UniqueName: \"kubernetes.io/projected/b78d2fb4-8068-48fc-87b7-32b6a0086f79-kube-api-access-mdzfx\") pod \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") "
Sep 10 15:11:27.100928 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.100867    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-metrics-certs\") pod \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\" (UID: \"b78d2fb4-8068-48fc-87b7-32b6a0086f79\") "
Sep 10 15:11:27.101243 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.101212    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78d2fb4-8068-48fc-87b7-32b6a0086f79-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b78d2fb4-8068-48fc-87b7-32b6a0086f79" (UID: "b78d2fb4-8068-48fc-87b7-32b6a0086f79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:11:27.103077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.103050    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78d2fb4-8068-48fc-87b7-32b6a0086f79-kube-api-access-mdzfx" (OuterVolumeSpecName: "kube-api-access-mdzfx") pod "b78d2fb4-8068-48fc-87b7-32b6a0086f79" (UID: "b78d2fb4-8068-48fc-87b7-32b6a0086f79"). InnerVolumeSpecName "kube-api-access-mdzfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:11:27.103077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.103073    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b78d2fb4-8068-48fc-87b7-32b6a0086f79" (UID: "b78d2fb4-8068-48fc-87b7-32b6a0086f79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:11:27.103275 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.103146    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b78d2fb4-8068-48fc-87b7-32b6a0086f79" (UID: "b78d2fb4-8068-48fc-87b7-32b6a0086f79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:11:27.163137 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.163083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:27.202210 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.202174    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:27.202210 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.202198    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b78d2fb4-8068-48fc-87b7-32b6a0086f79-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:27.202210 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.202209    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdzfx\" (UniqueName: \"kubernetes.io/projected/b78d2fb4-8068-48fc-87b7-32b6a0086f79-kube-api-access-mdzfx\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:27.202210 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.202219    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b78d2fb4-8068-48fc-87b7-32b6a0086f79-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:27.239547 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.239485    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:27.385677 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.385645    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"]
Sep 10 15:11:27.388407 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.388385    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-3f81b7-797c497569-xztz7"]
Sep 10 15:11:27.545362 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:27.545327    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:28.032666 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:28.032632    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b78d2fb4-8068-48fc-87b7-32b6a0086f79" path="/var/lib/kubelet/pods/b78d2fb4-8068-48fc-87b7-32b6a0086f79/volumes"
Sep 10 15:11:31.174001 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:31.173963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:34.070097 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:34.070050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:34.173677 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:34.173641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:34.349981 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:34.349903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:11:36.916072 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:36.916036    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-2ghvs"]
Sep 10 15:11:36.916602 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:36.916237    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs" podUID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" containerName="manager" containerID="cri-o://3c0ff3ad6761ca3a7f3d8dabc50f5b0bc2840fc7ae2586c7a3e02e19acbb52d7" gracePeriod=30
Sep 10 15:11:37.120476 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.120439    2563 generic.go:356] "Generic (PLEG): container finished" podID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" containerID="3c0ff3ad6761ca3a7f3d8dabc50f5b0bc2840fc7ae2586c7a3e02e19acbb52d7" exitCode=0
Sep 10 15:11:37.120665 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.120489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs" event={"ID":"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1","Type":"ContainerDied","Data":"3c0ff3ad6761ca3a7f3d8dabc50f5b0bc2840fc7ae2586c7a3e02e19acbb52d7"}
Sep 10 15:11:37.120665 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.120587    2563 scope.go:117] "RemoveContainer" containerID="149beb688e7955d3c331331bdd4ceb6d8ad050e92f7242970e452ab3e45aac3a"
Sep 10 15:11:37.165394 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.165370    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 15:11:37.290323 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.290233    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-webhook-cert\") pod \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") "
Sep 10 15:11:37.290323 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.290317    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-metrics-certs\") pod \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") "
Sep 10 15:11:37.290594 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.290346    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8bjz\" (UniqueName: \"kubernetes.io/projected/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-kube-api-access-k8bjz\") pod \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\" (UID: \"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1\") "
Sep 10 15:11:37.292361 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.292327    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" (UID: "7ed2fdf5-3f55-4033-90c0-1fac2531d4e1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:11:37.292472 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.292367    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-kube-api-access-k8bjz" (OuterVolumeSpecName: "kube-api-access-k8bjz") pod "7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" (UID: "7ed2fdf5-3f55-4033-90c0-1fac2531d4e1"). InnerVolumeSpecName "kube-api-access-k8bjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:11:37.292472 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.292432    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" (UID: "7ed2fdf5-3f55-4033-90c0-1fac2531d4e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:11:37.391644 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.391612    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-webhook-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:37.391644 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.391641    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:37.391644 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:37.391651    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8bjz\" (UniqueName: \"kubernetes.io/projected/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1-kube-api-access-k8bjz\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:11:38.126410 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:38.126381    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs"
Sep 10 15:11:38.126861 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:38.126386    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-2ghvs" event={"ID":"7ed2fdf5-3f55-4033-90c0-1fac2531d4e1","Type":"ContainerDied","Data":"6d2579099cfe2cbcf03790d8ff167837ddecca4bf1a9789a3625f6902e87c76d"}
Sep 10 15:11:38.126861 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:38.126534    2563 scope.go:117] "RemoveContainer" containerID="3c0ff3ad6761ca3a7f3d8dabc50f5b0bc2840fc7ae2586c7a3e02e19acbb52d7"
Sep 10 15:11:38.142683 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:38.142656    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-2ghvs"]
Sep 10 15:11:38.149415 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:38.149393    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-2ghvs"]
Sep 10 15:11:40.031729 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:40.031677    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" path="/var/lib/kubelet/pods/7ed2fdf5-3f55-4033-90c0-1fac2531d4e1/volumes"
Sep 10 15:11:45.099588 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:11:45.099547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:12:10.098551 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:10.098524    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 15:12:10.109873 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:10.109853    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 15:12:38.937002 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.936967    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-5djm2"]
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937330    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfd672e5-4573-411e-8e77-82ac6bfc7529" containerName="payload-processing"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937346    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfd672e5-4573-411e-8e77-82ac6bfc7529" containerName="payload-processing"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937355    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b78d2fb4-8068-48fc-87b7-32b6a0086f79" containerName="maas-api"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937361    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b78d2fb4-8068-48fc-87b7-32b6a0086f79" containerName="maas-api"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937377    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" containerName="manager"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937382    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" containerName="manager"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937389    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" containerName="manager"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937394    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" containerName="manager"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937401    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2edea59c-718b-4ce3-90a9-699f9f56aa68" containerName="payload-processing"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937408    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2edea59c-718b-4ce3-90a9-699f9f56aa68" containerName="payload-processing"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937420    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" containerName="maas-api"
Sep 10 15:12:38.937418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937426    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" containerName="maas-api"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937433    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b612830-9f63-4cb5-8b50-19d06ffa789f" containerName="payload-pre-processing"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937438    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b612830-9f63-4cb5-8b50-19d06ffa789f" containerName="payload-pre-processing"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937445    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937450    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937456    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937460    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937467    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937472    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937477    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937484    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937495    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71c5e681-023c-47a3-a857-4f4f51cdb7b2" containerName="maas-api"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937502    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="71c5e681-023c-47a3-a857-4f4f51cdb7b2" containerName="maas-api"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937526    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371dc4c1-a7e3-41d3-8197-f67b8c07a394" containerName="payload-pre-processing"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937533    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="371dc4c1-a7e3-41d3-8197-f67b8c07a394" containerName="payload-pre-processing"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937617    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="71c5e681-023c-47a3-a857-4f4f51cdb7b2" containerName="maas-api"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937628    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e5c48ed-d1d0-4a72-9995-5b1afe3f6ce9" containerName="maas-api"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937635    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="371dc4c1-a7e3-41d3-8197-f67b8c07a394" containerName="payload-pre-processing"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937640    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2edea59c-718b-4ce3-90a9-699f9f56aa68" containerName="payload-processing"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937646    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937650    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937656    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfd672e5-4573-411e-8e77-82ac6bfc7529" containerName="payload-processing"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937661    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="998f4479-b372-4c83-b1da-e6c71bc16d70" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937668    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" containerName="manager"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937673    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b78d2fb4-8068-48fc-87b7-32b6a0086f79" containerName="maas-api"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937677    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c8df1d4-b609-4564-9c1b-59ad81270c15" containerName="istio-proxy"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937682    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b612830-9f63-4cb5-8b50-19d06ffa789f" containerName="payload-pre-processing"
Sep 10 15:12:38.937854 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.937687    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ed2fdf5-3f55-4033-90c0-1fac2531d4e1" containerName="manager"
Sep 10 15:12:38.941245 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.941228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:38.943693 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.943671    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 15:12:38.943693 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.943692    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 15:12:38.943869 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.943690    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhrqr\""
Sep 10 15:12:38.943869 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.943748    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 15:12:38.950965 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:38.950945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-5djm2"]
Sep 10 15:12:39.035725 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.035694    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-webhook-cert\") pod \"maas-controller-5bcc5b4d4d-5djm2\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") " pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.036025 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.035800    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-metrics-certs\") pod \"maas-controller-5bcc5b4d4d-5djm2\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") " pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.036025 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.035825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg9c5\" (UniqueName: \"kubernetes.io/projected/65b15411-3f48-4cad-b653-4ac0f051bb15-kube-api-access-lg9c5\") pod \"maas-controller-5bcc5b4d4d-5djm2\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") " pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.136668 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.136628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-metrics-certs\") pod \"maas-controller-5bcc5b4d4d-5djm2\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") " pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.136884 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.136672    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg9c5\" (UniqueName: \"kubernetes.io/projected/65b15411-3f48-4cad-b653-4ac0f051bb15-kube-api-access-lg9c5\") pod \"maas-controller-5bcc5b4d4d-5djm2\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") " pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.136884 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.136748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-webhook-cert\") pod \"maas-controller-5bcc5b4d4d-5djm2\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") " pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.139273 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.139249    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-webhook-cert\") pod \"maas-controller-5bcc5b4d4d-5djm2\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") " pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.139393 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.139307    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-metrics-certs\") pod \"maas-controller-5bcc5b4d4d-5djm2\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") " pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.144820 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.144792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg9c5\" (UniqueName: \"kubernetes.io/projected/65b15411-3f48-4cad-b653-4ac0f051bb15-kube-api-access-lg9c5\") pod \"maas-controller-5bcc5b4d4d-5djm2\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") " pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.251708 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.251620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:39.523023 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.522988    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-5djm2"]
Sep 10 15:12:39.525316 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:12:39.525290    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b15411_3f48_4cad_b653_4ac0f051bb15.slice/crio-2265bfff8848343c89e8b587ac704709d3812d87383458816b5029df6123fb6d WatchSource:0}: Error finding container 2265bfff8848343c89e8b587ac704709d3812d87383458816b5029df6123fb6d: Status 404 returned error can't find the container with id 2265bfff8848343c89e8b587ac704709d3812d87383458816b5029df6123fb6d
Sep 10 15:12:39.526821 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:39.526799    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:12:40.422196 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:40.422160    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2" event={"ID":"65b15411-3f48-4cad-b653-4ac0f051bb15","Type":"ContainerStarted","Data":"a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe"}
Sep 10 15:12:40.422709 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:40.422219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2" event={"ID":"65b15411-3f48-4cad-b653-4ac0f051bb15","Type":"ContainerStarted","Data":"2265bfff8848343c89e8b587ac704709d3812d87383458816b5029df6123fb6d"}
Sep 10 15:12:40.422709 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:40.422261    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:12:40.440815 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:40.440768    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2" podStartSLOduration=2.049136573 podStartE2EDuration="2.440751673s" totalImagesPullingTime="391.6151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:12:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:12:40.439600995 +0000 UTC m=+1830.916660940" watchObservedRunningTime="2026-09-10 15:12:40.440751673 +0000 UTC m=+1830.917811613"
Sep 10 15:12:51.429016 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:12:51.428938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:13:00.610713 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:00.610677    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:13:02.804653 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:02.804609    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:13:02.898837 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:02.898805    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-5djm2"]
Sep 10 15:13:02.899083 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:02.899053    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2" podUID="65b15411-3f48-4cad-b653-4ac0f051bb15" containerName="manager" containerID="cri-o://a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe" gracePeriod=30
Sep 10 15:13:03.149405 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.149379    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:13:03.250588 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.250561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg9c5\" (UniqueName: \"kubernetes.io/projected/65b15411-3f48-4cad-b653-4ac0f051bb15-kube-api-access-lg9c5\") pod \"65b15411-3f48-4cad-b653-4ac0f051bb15\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") "
Sep 10 15:13:03.250781 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.250632    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-webhook-cert\") pod \"65b15411-3f48-4cad-b653-4ac0f051bb15\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") "
Sep 10 15:13:03.250781 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.250672    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-metrics-certs\") pod \"65b15411-3f48-4cad-b653-4ac0f051bb15\" (UID: \"65b15411-3f48-4cad-b653-4ac0f051bb15\") "
Sep 10 15:13:03.252643 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.252613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b15411-3f48-4cad-b653-4ac0f051bb15-kube-api-access-lg9c5" (OuterVolumeSpecName: "kube-api-access-lg9c5") pod "65b15411-3f48-4cad-b653-4ac0f051bb15" (UID: "65b15411-3f48-4cad-b653-4ac0f051bb15"). InnerVolumeSpecName "kube-api-access-lg9c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:13:03.252760 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.252687    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65b15411-3f48-4cad-b653-4ac0f051bb15" (UID: "65b15411-3f48-4cad-b653-4ac0f051bb15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:13:03.252760 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.252721    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "65b15411-3f48-4cad-b653-4ac0f051bb15" (UID: "65b15411-3f48-4cad-b653-4ac0f051bb15"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:13:03.351569 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.351484    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg9c5\" (UniqueName: \"kubernetes.io/projected/65b15411-3f48-4cad-b653-4ac0f051bb15-kube-api-access-lg9c5\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:13:03.351569 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.351536    2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-webhook-cert\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:13:03.351569 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.351552    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b15411-3f48-4cad-b653-4ac0f051bb15-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:13:03.537074 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.537042    2563 generic.go:356] "Generic (PLEG): container finished" podID="65b15411-3f48-4cad-b653-4ac0f051bb15" containerID="a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe" exitCode=0
Sep 10 15:13:03.537239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.537090    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2"
Sep 10 15:13:03.537239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.537126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2" event={"ID":"65b15411-3f48-4cad-b653-4ac0f051bb15","Type":"ContainerDied","Data":"a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe"}
Sep 10 15:13:03.537239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.537181    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-5djm2" event={"ID":"65b15411-3f48-4cad-b653-4ac0f051bb15","Type":"ContainerDied","Data":"2265bfff8848343c89e8b587ac704709d3812d87383458816b5029df6123fb6d"}
Sep 10 15:13:03.537239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.537205    2563 scope.go:117] "RemoveContainer" containerID="a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe"
Sep 10 15:13:03.546924 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.546897    2563 scope.go:117] "RemoveContainer" containerID="a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe"
Sep 10 15:13:03.547211 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:13:03.547189    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe\": container with ID starting with a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe not found: ID does not exist" containerID="a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe"
Sep 10 15:13:03.547267 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.547218    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe"} err="failed to get container status \"a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe\": rpc error: code = NotFound desc = could not find container \"a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe\": container with ID starting with a343fe97f06eefd78fc4b3a6142fc39f09077f9865c00af1ebd2cd2b157396fe not found: ID does not exist"
Sep 10 15:13:03.559904 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.559873    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-5djm2"]
Sep 10 15:13:03.567139 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:03.567114    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-5djm2"]
Sep 10 15:13:04.031802 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.031770    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b15411-3f48-4cad-b653-4ac0f051bb15" path="/var/lib/kubelet/pods/65b15411-3f48-4cad-b653-4ac0f051bb15/volumes"
Sep 10 15:13:04.252865 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.252827    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:13:04.360641 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.360552    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-2h97w"]
Sep 10 15:13:04.361000 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.360983    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b15411-3f48-4cad-b653-4ac0f051bb15" containerName="manager"
Sep 10 15:13:04.361092 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.361005    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b15411-3f48-4cad-b653-4ac0f051bb15" containerName="manager"
Sep 10 15:13:04.361146 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.361096    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b15411-3f48-4cad-b653-4ac0f051bb15" containerName="manager"
Sep 10 15:13:04.365404 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.365380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.368152 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.368128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 15:13:04.368270 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.368195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mhrqr\""
Sep 10 15:13:04.368270 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.368135    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 15:13:04.368396 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.368290    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 15:13:04.372831 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.372808    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-2h97w"]
Sep 10 15:13:04.463240 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.463191    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/071994e7-4084-4a07-b5af-c6426571c6e3-metrics-certs\") pod \"maas-controller-5bcc5b4d4d-2h97w\" (UID: \"071994e7-4084-4a07-b5af-c6426571c6e3\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.463415 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.463251    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/071994e7-4084-4a07-b5af-c6426571c6e3-webhook-cert\") pod \"maas-controller-5bcc5b4d4d-2h97w\" (UID: \"071994e7-4084-4a07-b5af-c6426571c6e3\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.463415 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.463373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrsqw\" (UniqueName: \"kubernetes.io/projected/071994e7-4084-4a07-b5af-c6426571c6e3-kube-api-access-rrsqw\") pod \"maas-controller-5bcc5b4d4d-2h97w\" (UID: \"071994e7-4084-4a07-b5af-c6426571c6e3\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.564859 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.564824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrsqw\" (UniqueName: \"kubernetes.io/projected/071994e7-4084-4a07-b5af-c6426571c6e3-kube-api-access-rrsqw\") pod \"maas-controller-5bcc5b4d4d-2h97w\" (UID: \"071994e7-4084-4a07-b5af-c6426571c6e3\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.565066 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.564882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/071994e7-4084-4a07-b5af-c6426571c6e3-metrics-certs\") pod \"maas-controller-5bcc5b4d4d-2h97w\" (UID: \"071994e7-4084-4a07-b5af-c6426571c6e3\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.565066 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.564916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/071994e7-4084-4a07-b5af-c6426571c6e3-webhook-cert\") pod \"maas-controller-5bcc5b4d4d-2h97w\" (UID: \"071994e7-4084-4a07-b5af-c6426571c6e3\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.567435 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.567410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/071994e7-4084-4a07-b5af-c6426571c6e3-webhook-cert\") pod \"maas-controller-5bcc5b4d4d-2h97w\" (UID: \"071994e7-4084-4a07-b5af-c6426571c6e3\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.567564 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.567460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/071994e7-4084-4a07-b5af-c6426571c6e3-metrics-certs\") pod \"maas-controller-5bcc5b4d4d-2h97w\" (UID: \"071994e7-4084-4a07-b5af-c6426571c6e3\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.572810 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.572793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrsqw\" (UniqueName: \"kubernetes.io/projected/071994e7-4084-4a07-b5af-c6426571c6e3-kube-api-access-rrsqw\") pod \"maas-controller-5bcc5b4d4d-2h97w\" (UID: \"071994e7-4084-4a07-b5af-c6426571c6e3\") " pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.675963 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.675929    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:04.948602 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:04.948491    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5bcc5b4d4d-2h97w"]
Sep 10 15:13:04.951955 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:13:04.951929    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod071994e7_4084_4a07_b5af_c6426571c6e3.slice/crio-332ecf982bc0d81cf8e32bd1e310033174664e36088209b0e67c463059fb6b50 WatchSource:0}: Error finding container 332ecf982bc0d81cf8e32bd1e310033174664e36088209b0e67c463059fb6b50: Status 404 returned error can't find the container with id 332ecf982bc0d81cf8e32bd1e310033174664e36088209b0e67c463059fb6b50
Sep 10 15:13:05.550568 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:05.550534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w" event={"ID":"071994e7-4084-4a07-b5af-c6426571c6e3","Type":"ContainerStarted","Data":"090bd2be9e0f065a2c52fc7c9496ca3e523046ff0520d08a40e9b3bb1bbdcfcf"}
Sep 10 15:13:05.550934 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:05.550585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w" event={"ID":"071994e7-4084-4a07-b5af-c6426571c6e3","Type":"ContainerStarted","Data":"332ecf982bc0d81cf8e32bd1e310033174664e36088209b0e67c463059fb6b50"}
Sep 10 15:13:05.550934 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:05.550665    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:13:05.567293 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:05.567251    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w" podStartSLOduration=1.061397093 podStartE2EDuration="1.567236364s" totalImagesPullingTime="505.839271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:13:05.566145158 +0000 UTC m=+1856.043205115" watchObservedRunningTime="2026-09-10 15:13:05.567236364 +0000 UTC m=+1856.044296309"
Sep 10 15:13:16.557870 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:13:16.557831    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5bcc5b4d4d-2h97w"
Sep 10 15:14:09.708963 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:09.708930    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"]
Sep 10 15:14:09.709488 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:09.709151    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn" podUID="4052641e-da82-44a8-9f3f-6459c0c324c7" containerName="manager" containerID="cri-o://0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a" gracePeriod=10
Sep 10 15:14:11.851635 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.851611    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 15:14:11.888079 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.888043    2563 generic.go:356] "Generic (PLEG): container finished" podID="4052641e-da82-44a8-9f3f-6459c0c324c7" containerID="0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a" exitCode=0
Sep 10 15:14:11.888079 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.888081    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"
Sep 10 15:14:11.888328 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.888080    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn" event={"ID":"4052641e-da82-44a8-9f3f-6459c0c324c7","Type":"ContainerDied","Data":"0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a"}
Sep 10 15:14:11.888328 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.888139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn" event={"ID":"4052641e-da82-44a8-9f3f-6459c0c324c7","Type":"ContainerDied","Data":"5a96864119c1099a92cf6cb39cc19731705c2c04aa9008e323c72c08d928e860"}
Sep 10 15:14:11.888328 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.888166    2563 scope.go:117] "RemoveContainer" containerID="0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a"
Sep 10 15:14:11.897248 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.897228    2563 scope.go:117] "RemoveContainer" containerID="0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a"
Sep 10 15:14:11.897611 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:14:11.897591    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a\": container with ID starting with 0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a not found: ID does not exist" containerID="0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a"
Sep 10 15:14:11.897701 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.897638    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a"} err="failed to get container status \"0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a\": rpc error: code = NotFound desc = could not find container \"0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a\": container with ID starting with 0a6816e00ddb11f8298e408944c32b44e3ee00ca00101ae28decfad363e9e36a not found: ID does not exist"
Sep 10 15:14:11.947576 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.947546    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4052641e-da82-44a8-9f3f-6459c0c324c7-extensions-socket-volume\") pod \"4052641e-da82-44a8-9f3f-6459c0c324c7\" (UID: \"4052641e-da82-44a8-9f3f-6459c0c324c7\") "
Sep 10 15:14:11.947736 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.947595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkhzq\" (UniqueName: \"kubernetes.io/projected/4052641e-da82-44a8-9f3f-6459c0c324c7-kube-api-access-jkhzq\") pod \"4052641e-da82-44a8-9f3f-6459c0c324c7\" (UID: \"4052641e-da82-44a8-9f3f-6459c0c324c7\") "
Sep 10 15:14:11.947931 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.947908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4052641e-da82-44a8-9f3f-6459c0c324c7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4052641e-da82-44a8-9f3f-6459c0c324c7" (UID: "4052641e-da82-44a8-9f3f-6459c0c324c7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:14:11.949670 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:11.949642    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4052641e-da82-44a8-9f3f-6459c0c324c7-kube-api-access-jkhzq" (OuterVolumeSpecName: "kube-api-access-jkhzq") pod "4052641e-da82-44a8-9f3f-6459c0c324c7" (UID: "4052641e-da82-44a8-9f3f-6459c0c324c7"). InnerVolumeSpecName "kube-api-access-jkhzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:14:12.048741 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:12.048718    2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4052641e-da82-44a8-9f3f-6459c0c324c7-extensions-socket-volume\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:14:12.048741 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:12.048742    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkhzq\" (UniqueName: \"kubernetes.io/projected/4052641e-da82-44a8-9f3f-6459c0c324c7-kube-api-access-jkhzq\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:14:12.205851 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:12.205815    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"]
Sep 10 15:14:12.212583 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:12.212555    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-82wbn"]
Sep 10 15:14:14.034035 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:14.034004    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4052641e-da82-44a8-9f3f-6459c0c324c7" path="/var/lib/kubelet/pods/4052641e-da82-44a8-9f3f-6459c0c324c7/volumes"
Sep 10 15:14:45.202153 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:45.202110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:14:45.288803 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:14:45.288772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:15:10.414738 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.414706    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"]
Sep 10 15:15:10.415119 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.415057    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4052641e-da82-44a8-9f3f-6459c0c324c7" containerName="manager"
Sep 10 15:15:10.415119 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.415067    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4052641e-da82-44a8-9f3f-6459c0c324c7" containerName="manager"
Sep 10 15:15:10.415202 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.415137    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4052641e-da82-44a8-9f3f-6459c0c324c7" containerName="manager"
Sep 10 15:15:10.419095 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.419080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:10.422651 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.422628    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wzp2t\""
Sep 10 15:15:10.430632 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.430612    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"]
Sep 10 15:15:10.554288 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.554247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh7fv\" (UniqueName: \"kubernetes.io/projected/d10f8dc3-32d1-482f-a2a6-7ce2adc357b6-kube-api-access-wh7fv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6lcn5\" (UID: \"d10f8dc3-32d1-482f-a2a6-7ce2adc357b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:10.554468 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.554336    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d10f8dc3-32d1-482f-a2a6-7ce2adc357b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6lcn5\" (UID: \"d10f8dc3-32d1-482f-a2a6-7ce2adc357b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:10.655614 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.655584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7fv\" (UniqueName: \"kubernetes.io/projected/d10f8dc3-32d1-482f-a2a6-7ce2adc357b6-kube-api-access-wh7fv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6lcn5\" (UID: \"d10f8dc3-32d1-482f-a2a6-7ce2adc357b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:10.655812 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.655642    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d10f8dc3-32d1-482f-a2a6-7ce2adc357b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6lcn5\" (UID: \"d10f8dc3-32d1-482f-a2a6-7ce2adc357b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:10.655990 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.655974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d10f8dc3-32d1-482f-a2a6-7ce2adc357b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6lcn5\" (UID: \"d10f8dc3-32d1-482f-a2a6-7ce2adc357b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:10.663804 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.663777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh7fv\" (UniqueName: \"kubernetes.io/projected/d10f8dc3-32d1-482f-a2a6-7ce2adc357b6-kube-api-access-wh7fv\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6lcn5\" (UID: \"d10f8dc3-32d1-482f-a2a6-7ce2adc357b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:10.729228 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:10.729190    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:11.009809 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:11.009783    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"]
Sep 10 15:15:11.011777 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:15:11.011747    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd10f8dc3_32d1_482f_a2a6_7ce2adc357b6.slice/crio-0c6a544fb26f1c7e75939a2c2d563b3c8141c16e932f6c14b091a68cac83f0c3 WatchSource:0}: Error finding container 0c6a544fb26f1c7e75939a2c2d563b3c8141c16e932f6c14b091a68cac83f0c3: Status 404 returned error can't find the container with id 0c6a544fb26f1c7e75939a2c2d563b3c8141c16e932f6c14b091a68cac83f0c3
Sep 10 15:15:11.165077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:11.165041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5" event={"ID":"d10f8dc3-32d1-482f-a2a6-7ce2adc357b6","Type":"ContainerStarted","Data":"92bb8fd3760f8b423864512980679587ecb871e0896395d209dd3189b0925e72"}
Sep 10 15:15:11.165259 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:11.165120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5" event={"ID":"d10f8dc3-32d1-482f-a2a6-7ce2adc357b6","Type":"ContainerStarted","Data":"0c6a544fb26f1c7e75939a2c2d563b3c8141c16e932f6c14b091a68cac83f0c3"}
Sep 10 15:15:11.165259 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:11.165160    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:11.186358 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:11.186120    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5" podStartSLOduration=1.186102148 podStartE2EDuration="1.186102148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:15:11.185346293 +0000 UTC m=+1981.662406236" watchObservedRunningTime="2026-09-10 15:15:11.186102148 +0000 UTC m=+1981.663162092"
Sep 10 15:15:22.172977 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:22.172939    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6lcn5"
Sep 10 15:15:26.830931 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:15:26.830889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:16:00.392796 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:00.392754    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:16:05.983462 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:05.983427    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"]
Sep 10 15:16:05.987048 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:05.987027    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:05.989671 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:05.989650    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7c0e79df-kserve-self-signed-certs\""
Sep 10 15:16:05.999334 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:05.999311    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"]
Sep 10 15:16:06.017594 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.017567    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:16:06.132992 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.132951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hnxv\" (UniqueName: \"kubernetes.io/projected/53899e97-10eb-4719-b941-789bff500217-kube-api-access-8hnxv\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.133191 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.133027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-model-cache\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.133191 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.133124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-home\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.133191 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.133154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53899e97-10eb-4719-b941-789bff500217-tls-certs\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.133369 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.133217    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-dshm\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.133369 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.133274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-tmp-dir\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.234555 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.234457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53899e97-10eb-4719-b941-789bff500217-tls-certs\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.234555 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.234496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-dshm\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.234555 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.234538    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-tmp-dir\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.234834 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.234591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnxv\" (UniqueName: \"kubernetes.io/projected/53899e97-10eb-4719-b941-789bff500217-kube-api-access-8hnxv\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.234834 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.234623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-model-cache\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.234834 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.234681    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-home\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.235076 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.235049    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-model-cache\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.235158 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.235073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-tmp-dir\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.235158 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.235129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-home\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.236805 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.236783    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-dshm\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.237075 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.237055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53899e97-10eb-4719-b941-789bff500217-tls-certs\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.243408 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.243384    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnxv\" (UniqueName: \"kubernetes.io/projected/53899e97-10eb-4719-b941-789bff500217-kube-api-access-8hnxv\") pod \"e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") " pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.295879 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.295843    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:06.571749 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:06.571725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"]
Sep 10 15:16:06.573864 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:16:06.573824    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53899e97_10eb_4719_b941_789bff500217.slice/crio-3b7fe94b96d0f8fcbe516e9b53e0bc365f5af582f96dc08db528af4e19e7cac5 WatchSource:0}: Error finding container 3b7fe94b96d0f8fcbe516e9b53e0bc365f5af582f96dc08db528af4e19e7cac5: Status 404 returned error can't find the container with id 3b7fe94b96d0f8fcbe516e9b53e0bc365f5af582f96dc08db528af4e19e7cac5
Sep 10 15:16:07.429909 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:07.429872    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj" event={"ID":"53899e97-10eb-4719-b941-789bff500217","Type":"ContainerStarted","Data":"c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5"}
Sep 10 15:16:07.429909 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:07.429914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj" event={"ID":"53899e97-10eb-4719-b941-789bff500217","Type":"ContainerStarted","Data":"3b7fe94b96d0f8fcbe516e9b53e0bc365f5af582f96dc08db528af4e19e7cac5"}
Sep 10 15:16:07.447052 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:07.447004    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj" podStartSLOduration=2.446990284 podStartE2EDuration="2.446990284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:07.446194299 +0000 UTC m=+2037.923254243" watchObservedRunningTime="2026-09-10 15:16:07.446990284 +0000 UTC m=+2037.924050227"
Sep 10 15:16:16.296577 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:16.296486    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:16.296976 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:16.296605    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:16.309602 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:16.309575    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:16.482610 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:16.482575    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:19.869452 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:19.869416    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"]
Sep 10 15:16:19.873077 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:19.873052    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:19.875028 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:19.875009    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-7c0e79df-kserve-self-signed-certs\""
Sep 10 15:16:19.882968 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:19.882945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"]
Sep 10 15:16:19.963006 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:19.962958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:16:20.057976 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.057947    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-tmp-dir\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.058136 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.057996    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-dshm\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.058136 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.058044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffda527d-280e-4ae1-995d-765b3f68e3a4-tls-certs\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.058136 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.058112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-model-cache\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.058263 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.058147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-home\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.058263 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.058169    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjx7\" (UniqueName: \"kubernetes.io/projected/ffda527d-280e-4ae1-995d-765b3f68e3a4-kube-api-access-vmjx7\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.159167 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.159139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-home\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.159321 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.159176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjx7\" (UniqueName: \"kubernetes.io/projected/ffda527d-280e-4ae1-995d-765b3f68e3a4-kube-api-access-vmjx7\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.159321 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.159224    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-tmp-dir\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.159321 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.159268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-dshm\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.159321 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.159289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffda527d-280e-4ae1-995d-765b3f68e3a4-tls-certs\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.159470 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.159335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-model-cache\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.159631 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.159607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-home\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.159677 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.159628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-tmp-dir\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.159794 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.159778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-model-cache\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.161578 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.161554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-dshm\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.161821 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.161800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffda527d-280e-4ae1-995d-765b3f68e3a4-tls-certs\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.167138 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.167119    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjx7\" (UniqueName: \"kubernetes.io/projected/ffda527d-280e-4ae1-995d-765b3f68e3a4-kube-api-access-vmjx7\") pod \"e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") " pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.183019 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.182998    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:20.460111 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.460088    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"]
Sep 10 15:16:20.462246 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:16:20.462209    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffda527d_280e_4ae1_995d_765b3f68e3a4.slice/crio-6fa331812293b4cce264b11499d7541f941293f632804b89b19f3bf2ea9fa009 WatchSource:0}: Error finding container 6fa331812293b4cce264b11499d7541f941293f632804b89b19f3bf2ea9fa009: Status 404 returned error can't find the container with id 6fa331812293b4cce264b11499d7541f941293f632804b89b19f3bf2ea9fa009
Sep 10 15:16:20.491143 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:20.491116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g" event={"ID":"ffda527d-280e-4ae1-995d-765b3f68e3a4","Type":"ContainerStarted","Data":"6fa331812293b4cce264b11499d7541f941293f632804b89b19f3bf2ea9fa009"}
Sep 10 15:16:21.498521 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:21.498466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g" event={"ID":"ffda527d-280e-4ae1-995d-765b3f68e3a4","Type":"ContainerStarted","Data":"dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789"}
Sep 10 15:16:21.517050 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:21.516986    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g" podStartSLOduration=2.5169448709999998 podStartE2EDuration="2.516944871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:21.514977685 +0000 UTC m=+2051.992037629" watchObservedRunningTime="2026-09-10 15:16:21.516944871 +0000 UTC m=+2051.994004817"
Sep 10 15:16:30.183272 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:30.183239    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:30.183658 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:30.183288    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:30.196075 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:30.196052    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:30.554919 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:30.554841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:32.819621 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:32.819541    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"]
Sep 10 15:16:32.918635 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:32.918597    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:16:34.330351 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:34.330317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"]
Sep 10 15:16:34.351689 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:34.351657    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:16:45.323258 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:45.323219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:16:47.839543 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:47.839454    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj" podUID="53899e97-10eb-4719-b941-789bff500217" containerName="main" containerID="cri-o://c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5" gracePeriod=45
Sep 10 15:16:48.090813 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.090741    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:48.203975 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.203945    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53899e97-10eb-4719-b941-789bff500217-tls-certs\") pod \"53899e97-10eb-4719-b941-789bff500217\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") "
Sep 10 15:16:48.204178 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-model-cache\") pod \"53899e97-10eb-4719-b941-789bff500217\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") "
Sep 10 15:16:48.204178 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204052    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hnxv\" (UniqueName: \"kubernetes.io/projected/53899e97-10eb-4719-b941-789bff500217-kube-api-access-8hnxv\") pod \"53899e97-10eb-4719-b941-789bff500217\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") "
Sep 10 15:16:48.204178 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-home\") pod \"53899e97-10eb-4719-b941-789bff500217\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") "
Sep 10 15:16:48.204178 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204103    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-tmp-dir\") pod \"53899e97-10eb-4719-b941-789bff500217\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") "
Sep 10 15:16:48.204178 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204165    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-dshm\") pod \"53899e97-10eb-4719-b941-789bff500217\" (UID: \"53899e97-10eb-4719-b941-789bff500217\") "
Sep 10 15:16:48.204426 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-model-cache" (OuterVolumeSpecName: "model-cache") pod "53899e97-10eb-4719-b941-789bff500217" (UID: "53899e97-10eb-4719-b941-789bff500217"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:16:48.204496 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-home" (OuterVolumeSpecName: "home") pod "53899e97-10eb-4719-b941-789bff500217" (UID: "53899e97-10eb-4719-b941-789bff500217"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:16:48.204592 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204542    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-model-cache\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:48.204592 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53899e97-10eb-4719-b941-789bff500217" (UID: "53899e97-10eb-4719-b941-789bff500217"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:16:48.204592 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.204560    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-home\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:48.206239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.206212    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-dshm" (OuterVolumeSpecName: "dshm") pod "53899e97-10eb-4719-b941-789bff500217" (UID: "53899e97-10eb-4719-b941-789bff500217"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:16:48.206239 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.206222    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53899e97-10eb-4719-b941-789bff500217-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53899e97-10eb-4719-b941-789bff500217" (UID: "53899e97-10eb-4719-b941-789bff500217"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:16:48.206384 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.206276    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53899e97-10eb-4719-b941-789bff500217-kube-api-access-8hnxv" (OuterVolumeSpecName: "kube-api-access-8hnxv") pod "53899e97-10eb-4719-b941-789bff500217" (UID: "53899e97-10eb-4719-b941-789bff500217"). InnerVolumeSpecName "kube-api-access-8hnxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:16:48.305418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.305381    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hnxv\" (UniqueName: \"kubernetes.io/projected/53899e97-10eb-4719-b941-789bff500217-kube-api-access-8hnxv\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:48.305418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.305410    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-tmp-dir\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:48.305418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.305423    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53899e97-10eb-4719-b941-789bff500217-dshm\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:48.305418 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.305430    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53899e97-10eb-4719-b941-789bff500217-tls-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:48.636698 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.636664    2563 generic.go:356] "Generic (PLEG): container finished" podID="53899e97-10eb-4719-b941-789bff500217" containerID="c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5" exitCode=0
Sep 10 15:16:48.636916 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.636705    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"
Sep 10 15:16:48.636916 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.636720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj" event={"ID":"53899e97-10eb-4719-b941-789bff500217","Type":"ContainerDied","Data":"c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5"}
Sep 10 15:16:48.636916 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.636762    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj" event={"ID":"53899e97-10eb-4719-b941-789bff500217","Type":"ContainerDied","Data":"3b7fe94b96d0f8fcbe516e9b53e0bc365f5af582f96dc08db528af4e19e7cac5"}
Sep 10 15:16:48.636916 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.636782    2563 scope.go:117] "RemoveContainer" containerID="c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5"
Sep 10 15:16:48.645985 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.645967    2563 scope.go:117] "RemoveContainer" containerID="c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5"
Sep 10 15:16:48.646254 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:16:48.646233    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5\": container with ID starting with c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5 not found: ID does not exist" containerID="c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5"
Sep 10 15:16:48.646321 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.646262    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5"} err="failed to get container status \"c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5\": rpc error: code = NotFound desc = could not find container \"c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5\": container with ID starting with c393eb6f4bd2def419cd1940e3e7dd5e1db12eabd3b197a6ca4dd21b2c9d78d5 not found: ID does not exist"
Sep 10 15:16:48.657030 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.657008    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"]
Sep 10 15:16:48.659422 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:48.659400    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7c0e79df-kserve-86f85c5bf9-p5cvj"]
Sep 10 15:16:49.351951 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.351909    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g" podUID="ffda527d-280e-4ae1-995d-765b3f68e3a4" containerName="main" containerID="cri-o://dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789" gracePeriod=45
Sep 10 15:16:49.608632 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.608572    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:49.644391 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.644359    2563 generic.go:356] "Generic (PLEG): container finished" podID="ffda527d-280e-4ae1-995d-765b3f68e3a4" containerID="dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789" exitCode=0
Sep 10 15:16:49.644610 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.644400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g" event={"ID":"ffda527d-280e-4ae1-995d-765b3f68e3a4","Type":"ContainerDied","Data":"dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789"}
Sep 10 15:16:49.644610 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.644416    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"
Sep 10 15:16:49.644610 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.644426    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g" event={"ID":"ffda527d-280e-4ae1-995d-765b3f68e3a4","Type":"ContainerDied","Data":"6fa331812293b4cce264b11499d7541f941293f632804b89b19f3bf2ea9fa009"}
Sep 10 15:16:49.644610 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.644447    2563 scope.go:117] "RemoveContainer" containerID="dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789"
Sep 10 15:16:49.652993 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.652970    2563 scope.go:117] "RemoveContainer" containerID="dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789"
Sep 10 15:16:49.653309 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:16:49.653290    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789\": container with ID starting with dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789 not found: ID does not exist" containerID="dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789"
Sep 10 15:16:49.653365 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.653318    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789"} err="failed to get container status \"dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789\": rpc error: code = NotFound desc = could not find container \"dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789\": container with ID starting with dd284f8d20d6c722d9b2837d68a76c3eae0f8eedcdd36e2e555d78e3228a7789 not found: ID does not exist"
Sep 10 15:16:49.717644 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.717615    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-model-cache\") pod \"ffda527d-280e-4ae1-995d-765b3f68e3a4\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") "
Sep 10 15:16:49.717644 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.717651    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-dshm\") pod \"ffda527d-280e-4ae1-995d-765b3f68e3a4\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") "
Sep 10 15:16:49.717912 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.717700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffda527d-280e-4ae1-995d-765b3f68e3a4-tls-certs\") pod \"ffda527d-280e-4ae1-995d-765b3f68e3a4\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") "
Sep 10 15:16:49.717912 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.717728    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-tmp-dir\") pod \"ffda527d-280e-4ae1-995d-765b3f68e3a4\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") "
Sep 10 15:16:49.717912 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.717755    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmjx7\" (UniqueName: \"kubernetes.io/projected/ffda527d-280e-4ae1-995d-765b3f68e3a4-kube-api-access-vmjx7\") pod \"ffda527d-280e-4ae1-995d-765b3f68e3a4\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") "
Sep 10 15:16:49.717912 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.717779    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-home\") pod \"ffda527d-280e-4ae1-995d-765b3f68e3a4\" (UID: \"ffda527d-280e-4ae1-995d-765b3f68e3a4\") "
Sep 10 15:16:49.718180 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.717921    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "ffda527d-280e-4ae1-995d-765b3f68e3a4" (UID: "ffda527d-280e-4ae1-995d-765b3f68e3a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:16:49.718180 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.718038    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ffda527d-280e-4ae1-995d-765b3f68e3a4" (UID: "ffda527d-280e-4ae1-995d-765b3f68e3a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:16:49.718180 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.718076    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-model-cache\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:49.718180 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.718103    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-home" (OuterVolumeSpecName: "home") pod "ffda527d-280e-4ae1-995d-765b3f68e3a4" (UID: "ffda527d-280e-4ae1-995d-765b3f68e3a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:16:49.720030 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.720005    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-dshm" (OuterVolumeSpecName: "dshm") pod "ffda527d-280e-4ae1-995d-765b3f68e3a4" (UID: "ffda527d-280e-4ae1-995d-765b3f68e3a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:16:49.720169 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.720075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffda527d-280e-4ae1-995d-765b3f68e3a4-kube-api-access-vmjx7" (OuterVolumeSpecName: "kube-api-access-vmjx7") pod "ffda527d-280e-4ae1-995d-765b3f68e3a4" (UID: "ffda527d-280e-4ae1-995d-765b3f68e3a4"). InnerVolumeSpecName "kube-api-access-vmjx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:16:49.720169 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.720090    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffda527d-280e-4ae1-995d-765b3f68e3a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ffda527d-280e-4ae1-995d-765b3f68e3a4" (UID: "ffda527d-280e-4ae1-995d-765b3f68e3a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:16:49.818919 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.818882    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffda527d-280e-4ae1-995d-765b3f68e3a4-tls-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:49.818919 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.818911    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-tmp-dir\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:49.818919 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.818922    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjx7\" (UniqueName: \"kubernetes.io/projected/ffda527d-280e-4ae1-995d-765b3f68e3a4-kube-api-access-vmjx7\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:49.819150 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.818930    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-home\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:49.819150 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.818940    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffda527d-280e-4ae1-995d-765b3f68e3a4-dshm\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:16:49.966405 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.966370    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"]
Sep 10 15:16:49.971611 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:49.971585    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-7c0e79df-kserve-859dbb64b7-6446g"]
Sep 10 15:16:50.032578 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:50.032546    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53899e97-10eb-4719-b941-789bff500217" path="/var/lib/kubelet/pods/53899e97-10eb-4719-b941-789bff500217/volumes"
Sep 10 15:16:50.032938 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:50.032926    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffda527d-280e-4ae1-995d-765b3f68e3a4" path="/var/lib/kubelet/pods/ffda527d-280e-4ae1-995d-765b3f68e3a4/volumes"
Sep 10 15:16:51.176217 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.176187    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"]
Sep 10 15:16:51.176588 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.176551    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffda527d-280e-4ae1-995d-765b3f68e3a4" containerName="main"
Sep 10 15:16:51.176588 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.176563    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffda527d-280e-4ae1-995d-765b3f68e3a4" containerName="main"
Sep 10 15:16:51.176588 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.176584    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53899e97-10eb-4719-b941-789bff500217" containerName="main"
Sep 10 15:16:51.176588 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.176592    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="53899e97-10eb-4719-b941-789bff500217" containerName="main"
Sep 10 15:16:51.176733 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.176650    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffda527d-280e-4ae1-995d-765b3f68e3a4" containerName="main"
Sep 10 15:16:51.176733 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.176662    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="53899e97-10eb-4719-b941-789bff500217" containerName="main"
Sep 10 15:16:51.181066 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.181044    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.187774 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.187747    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"]
Sep 10 15:16:51.333427 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.333389    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4k4\" (UniqueName: \"kubernetes.io/projected/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-kube-api-access-mm4k4\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.333601 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.333539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-maas-api-tls\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.333601 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.333580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-openshift-service-ca\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.333695 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.333641    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-metrics-certs\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.434931 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.434856    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-maas-api-tls\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.434931 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.434895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-openshift-service-ca\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.434931 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.434921    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-metrics-certs\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.435146 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.434954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4k4\" (UniqueName: \"kubernetes.io/projected/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-kube-api-access-mm4k4\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.435547 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.435498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-openshift-service-ca\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.437453 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.437430    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-maas-api-tls\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.437646 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.437624    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-metrics-certs\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.441807 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.441784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4k4\" (UniqueName: \"kubernetes.io/projected/87b25030-ae2d-4cbc-bb0b-281eca2bc3fd-kube-api-access-mm4k4\") pod \"maas-api-6d6c5d7bdb-hb6p5\" (UID: \"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd\") " pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.493914 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.493886    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:51.774136 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:51.774110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"]
Sep 10 15:16:51.776325 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:16:51.776295    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b25030_ae2d_4cbc_bb0b_281eca2bc3fd.slice/crio-37d52a81f934795f98bb35b5e42356b225e0727c7cf660ca4a3a125fea38c8d0 WatchSource:0}: Error finding container 37d52a81f934795f98bb35b5e42356b225e0727c7cf660ca4a3a125fea38c8d0: Status 404 returned error can't find the container with id 37d52a81f934795f98bb35b5e42356b225e0727c7cf660ca4a3a125fea38c8d0
Sep 10 15:16:52.662906 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:52.662874    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5" event={"ID":"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd","Type":"ContainerStarted","Data":"9511535b1a2763df673060edc9c7d17a9309642cda6fd7d3c669a4923870fb2f"}
Sep 10 15:16:52.663323 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:52.662928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5" event={"ID":"87b25030-ae2d-4cbc-bb0b-281eca2bc3fd","Type":"ContainerStarted","Data":"37d52a81f934795f98bb35b5e42356b225e0727c7cf660ca4a3a125fea38c8d0"}
Sep 10 15:16:52.663323 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:52.662966    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:16:52.681590 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:52.681549    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5" podStartSLOduration=1.290960211 podStartE2EDuration="1.681534531s" totalImagesPullingTime="390.57432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:16:52.680143869 +0000 UTC m=+2083.157203814" watchObservedRunningTime="2026-09-10 15:16:52.681534531 +0000 UTC m=+2083.158594468"
Sep 10 15:16:58.674013 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:16:58.673986    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d6c5d7bdb-hb6p5"
Sep 10 15:17:05.950324 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:05.950287    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"]
Sep 10 15:17:05.950888 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:05.950571    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2" podUID="053b0de1-9a92-4797-bfa6-d9653a09ee0e" containerName="maas-api" containerID="cri-o://8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac" gracePeriod=30
Sep 10 15:17:06.497387 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:06.497345    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2" podUID="053b0de1-9a92-4797-bfa6-d9653a09ee0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.40:8443/health\": dial tcp 10.132.0.40:8443: connect: connection refused"
Sep 10 15:17:07.061589 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:17:07.061558    2563 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053b0de1_9a92_4797_bfa6_d9653a09ee0e.slice/crio-8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 15:17:07.204659 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.204633    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 15:17:07.383311 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.383216    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053b0de1-9a92-4797-bfa6-d9653a09ee0e-openshift-service-ca\") pod \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") "
Sep 10 15:17:07.383311 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.383287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-metrics-certs\") pod \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") "
Sep 10 15:17:07.383311 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.383312    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-maas-api-tls\") pod \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") "
Sep 10 15:17:07.383633 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.383340    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfrc9\" (UniqueName: \"kubernetes.io/projected/053b0de1-9a92-4797-bfa6-d9653a09ee0e-kube-api-access-sfrc9\") pod \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\" (UID: \"053b0de1-9a92-4797-bfa6-d9653a09ee0e\") "
Sep 10 15:17:07.383714 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.383684    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053b0de1-9a92-4797-bfa6-d9653a09ee0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "053b0de1-9a92-4797-bfa6-d9653a09ee0e" (UID: "053b0de1-9a92-4797-bfa6-d9653a09ee0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:17:07.385448 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.385414    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "053b0de1-9a92-4797-bfa6-d9653a09ee0e" (UID: "053b0de1-9a92-4797-bfa6-d9653a09ee0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:17:07.385448 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.385438    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "053b0de1-9a92-4797-bfa6-d9653a09ee0e" (UID: "053b0de1-9a92-4797-bfa6-d9653a09ee0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:17:07.385680 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.385469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/053b0de1-9a92-4797-bfa6-d9653a09ee0e-kube-api-access-sfrc9" (OuterVolumeSpecName: "kube-api-access-sfrc9") pod "053b0de1-9a92-4797-bfa6-d9653a09ee0e" (UID: "053b0de1-9a92-4797-bfa6-d9653a09ee0e"). InnerVolumeSpecName "kube-api-access-sfrc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:17:07.484289 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.484258    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/053b0de1-9a92-4797-bfa6-d9653a09ee0e-openshift-service-ca\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:17:07.484289 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.484283    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-metrics-certs\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:17:07.484289 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.484293    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/053b0de1-9a92-4797-bfa6-d9653a09ee0e-maas-api-tls\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:17:07.484289 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.484301    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfrc9\" (UniqueName: \"kubernetes.io/projected/053b0de1-9a92-4797-bfa6-d9653a09ee0e-kube-api-access-sfrc9\") on node \"ip-10-0-141-186.ec2.internal\" DevicePath \"\""
Sep 10 15:17:07.739317 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.739282    2563 generic.go:356] "Generic (PLEG): container finished" podID="053b0de1-9a92-4797-bfa6-d9653a09ee0e" containerID="8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac" exitCode=0
Sep 10 15:17:07.739500 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.739333    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"
Sep 10 15:17:07.739500 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.739364    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2" event={"ID":"053b0de1-9a92-4797-bfa6-d9653a09ee0e","Type":"ContainerDied","Data":"8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac"}
Sep 10 15:17:07.739500 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.739406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2" event={"ID":"053b0de1-9a92-4797-bfa6-d9653a09ee0e","Type":"ContainerDied","Data":"d9551adda3bd2d552f860b516e1bccf21f6c65e8037b20d98f3a5190d4f315c7"}
Sep 10 15:17:07.739500 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.739427    2563 scope.go:117] "RemoveContainer" containerID="8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac"
Sep 10 15:17:07.749226 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.749209    2563 scope.go:117] "RemoveContainer" containerID="8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac"
Sep 10 15:17:07.749488 ip-10-0-141-186 kubenswrapper[2563]: E0910 15:17:07.749469    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac\": container with ID starting with 8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac not found: ID does not exist" containerID="8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac"
Sep 10 15:17:07.749569 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.749497    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac"} err="failed to get container status \"8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac\": rpc error: code = NotFound desc = could not find container \"8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac\": container with ID starting with 8c660fa51864edb0662b2e78362b87fd72d1e0c0a021e6250d7c2167649c59ac not found: ID does not exist"
Sep 10 15:17:07.765534 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.765485    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"]
Sep 10 15:17:07.772092 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:07.772065    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f975847c4-rjvs2"]
Sep 10 15:17:08.033840 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:08.033763    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="053b0de1-9a92-4797-bfa6-d9653a09ee0e" path="/var/lib/kubelet/pods/053b0de1-9a92-4797-bfa6-d9653a09ee0e/volumes"
Sep 10 15:17:10.127225 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:10.127193    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 15:17:10.140527 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:10.140482    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 15:17:26.492774 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:26.492742    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q692j_37acfe05-5214-4191-abb7-741d7d0e46b4/manager/1.log"
Sep 10 15:17:26.612007 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:26.611978    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ltrn4_f881eda7-2589-4f11-818b-12fbbdc66305/manager/0.log"
Sep 10 15:17:26.736877 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:26.736853    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bcc5b4d4d-2h97w_071994e7-4084-4a07-b5af-c6426571c6e3/manager/0.log"
Sep 10 15:17:27.361021 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:27.360993    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f84f89d7-mdzcx_b96e3a59-a1d6-4388-b780-7d0b6e4029db/copy-manifests/0.log"
Sep 10 15:17:27.374896 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:27.374873    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f84f89d7-mdzcx_b96e3a59-a1d6-4388-b780-7d0b6e4029db/manager/0.log"
Sep 10 15:17:27.705614 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:27.705580    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d6c5d7bdb-hb6p5_87b25030-ae2d-4cbc-bb0b-281eca2bc3fd/maas-api/0.log"
Sep 10 15:17:28.155937 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:28.155909    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kz8h5_0b1efb48-3e0d-4621-a8ca-3eddf45cfdf7/postgres/0.log"
Sep 10 15:17:29.030785 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:29.030746    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dvrf9_5f5e9277-b986-4595-8cbf-93da02b83a60/manager/0.log"
Sep 10 15:17:29.144425 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:29.144398    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-gq5m4_70d0014b-d871-4333-87f4-d8eec73976bc/manager/0.log"
Sep 10 15:17:29.377621 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:29.377542    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6lcn5_d10f8dc3-32d1-482f-a2a6-7ce2adc357b6/manager/0.log"
Sep 10 15:17:29.632164 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:29.632081    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-s86pg_372a2ece-1e88-4404-9651-5f7975f98587/manager/0.log"
Sep 10 15:17:30.219770 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:30.219742    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b7fd47cd-9h88c_52d2e320-c84e-4a03-a429-c25bcd89d238/kube-auth-proxy/0.log"
Sep 10 15:17:30.793910 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:30.793879    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59ccbd4756-2q8pl_f9b16477-9d87-43a7-9f53-2f5621126de6/router/0.log"
Sep 10 15:17:31.230100 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:31.230064    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-4zl47_13bd8514-e3ac-410b-8454-cd619c144491/main/0.log"
Sep 10 15:17:31.460575 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:31.460546    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-8vhkm_754462ae-275d-442d-950f-6539698e2440/main/0.log"
Sep 10 15:17:37.168252 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:37.168193    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:17:37.258751 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:37.258713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log" limit=1024
Sep 10 15:17:40.112075 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:40.112046    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-g9xjj_e30e502c-ff23-4cad-99f1-5c1b501b3fb4/global-pull-secret-syncer/0.log"
Sep 10 15:17:40.200969 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:40.200939    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ggpnx_5997abbb-1687-40dc-bdcf-84e52a57673a/konnectivity-agent/0.log"
Sep 10 15:17:40.318787 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:40.318761    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-186.ec2.internal_c8d04f9a962d39a0595afd41d5bd3b7a/haproxy/0.log"
Sep 10 15:17:45.093868 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:45.093837    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-dvrf9_5f5e9277-b986-4595-8cbf-93da02b83a60/manager/0.log"
Sep 10 15:17:45.130477 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:45.130451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-gq5m4_70d0014b-d871-4333-87f4-d8eec73976bc/manager/0.log"
Sep 10 15:17:45.199860 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:45.199786    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6lcn5_d10f8dc3-32d1-482f-a2a6-7ce2adc357b6/manager/0.log"
Sep 10 15:17:45.297980 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:45.297947    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-s86pg_372a2ece-1e88-4404-9651-5f7975f98587/manager/0.log"
Sep 10 15:17:46.928661 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:46.928629    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-f4qtm_9da82716-f9d8-443f-bb14-0552cd0c3a7a/cluster-monitoring-operator/0.log"
Sep 10 15:17:47.419355 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:47.419330    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bmp7j_b2eeea63-9b0f-4883-ae9a-9636e5ffe157/node-exporter/0.log"
Sep 10 15:17:47.459725 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:47.459702    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bmp7j_b2eeea63-9b0f-4883-ae9a-9636e5ffe157/kube-rbac-proxy/0.log"
Sep 10 15:17:47.489732 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:47.489712    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bmp7j_b2eeea63-9b0f-4883-ae9a-9636e5ffe157/init-textfile/0.log"
Sep 10 15:17:50.131103 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.131072    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/1.log"
Sep 10 15:17:50.135066 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.135044    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8wpzz_2a2b0239-057f-4348-8709-2406a8d82fae/console-operator/2.log"
Sep 10 15:17:50.310326 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.310291    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"]
Sep 10 15:17:50.310681 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.310668    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053b0de1-9a92-4797-bfa6-d9653a09ee0e" containerName="maas-api"
Sep 10 15:17:50.310746 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.310682    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="053b0de1-9a92-4797-bfa6-d9653a09ee0e" containerName="maas-api"
Sep 10 15:17:50.310789 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.310747    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="053b0de1-9a92-4797-bfa6-d9653a09ee0e" containerName="maas-api"
Sep 10 15:17:50.313830 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.313810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.315976 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.315951    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-prcmw\"/\"kube-root-ca.crt\""
Sep 10 15:17:50.316105 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.315990    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-prcmw\"/\"openshift-service-ca.crt\""
Sep 10 15:17:50.316105 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.316019    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-prcmw\"/\"default-dockercfg-9v44h\""
Sep 10 15:17:50.323100 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.323075    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"]
Sep 10 15:17:50.461011 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.460973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-lib-modules\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.461199 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.461088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-proc\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.461199 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.461110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-podres\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.461199 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.461128    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-sys\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.461199 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.461163    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4n6f\" (UniqueName: \"kubernetes.io/projected/30f0a089-409b-4d66-bd08-9997741cdd26-kube-api-access-j4n6f\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.562593 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.562544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-proc\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.562593 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.562602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-podres\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.562870 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.562628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-sys\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.562870 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.562658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-proc\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.562870 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.562668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4n6f\" (UniqueName: \"kubernetes.io/projected/30f0a089-409b-4d66-bd08-9997741cdd26-kube-api-access-j4n6f\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.562870 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.562728    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-lib-modules\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.562870 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.562750    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-sys\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.562870 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.562759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-podres\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.563076 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.562892    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/30f0a089-409b-4d66-bd08-9997741cdd26-lib-modules\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.570995 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.570970    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4n6f\" (UniqueName: \"kubernetes.io/projected/30f0a089-409b-4d66-bd08-9997741cdd26-kube-api-access-j4n6f\") pod \"perf-node-gather-daemonset-8l2gw\" (UID: \"30f0a089-409b-4d66-bd08-9997741cdd26\") " pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.625347 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.625325    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:50.896915 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.896890    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"]
Sep 10 15:17:50.899198 ip-10-0-141-186 kubenswrapper[2563]: W0910 15:17:50.899168    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod30f0a089_409b_4d66_bd08_9997741cdd26.slice/crio-11e3ce3fef9ae76343fb788817b313ee76e4e0c6b283c704833314572a75fa47 WatchSource:0}: Error finding container 11e3ce3fef9ae76343fb788817b313ee76e4e0c6b283c704833314572a75fa47: Status 404 returned error can't find the container with id 11e3ce3fef9ae76343fb788817b313ee76e4e0c6b283c704833314572a75fa47
Sep 10 15:17:50.956044 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:50.956019    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw" event={"ID":"30f0a089-409b-4d66-bd08-9997741cdd26","Type":"ContainerStarted","Data":"11e3ce3fef9ae76343fb788817b313ee76e4e0c6b283c704833314572a75fa47"}
Sep 10 15:17:51.442839 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:51.442811    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-jzbjz_052c643e-2396-4b74-893d-3b1c6f0689c1/volume-data-source-validator/0.log"
Sep 10 15:17:51.962075 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:51.962039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw" event={"ID":"30f0a089-409b-4d66-bd08-9997741cdd26","Type":"ContainerStarted","Data":"574ec8f70cb95ea4af6337f71832c6727e418d5ab1b32d28ea3716327f93e748"}
Sep 10 15:17:51.962260 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:51.962146    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:51.978397 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:51.978351    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw" podStartSLOduration=1.978337542 podStartE2EDuration="1.978337542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:17:51.976487877 +0000 UTC m=+2142.453547823" watchObservedRunningTime="2026-09-10 15:17:51.978337542 +0000 UTC m=+2142.455397486"
Sep 10 15:17:52.659360 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:52.659332    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mkrsh_0bab8655-a495-49a2-80e3-9a2177570f06/dns/0.log"
Sep 10 15:17:52.682114 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:52.682085    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mkrsh_0bab8655-a495-49a2-80e3-9a2177570f06/kube-rbac-proxy/0.log"
Sep 10 15:17:52.705269 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:52.705239    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-25clg_e47850cc-f2da-4235-8c44-db09fe594725/dns-node-resolver/0.log"
Sep 10 15:17:53.351108 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:53.351068    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c4tpn_e2f6a821-b6ef-46cd-8fb9-9116f71fa505/node-ca/0.log"
Sep 10 15:17:54.593090 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:54.593054    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b7fd47cd-9h88c_52d2e320-c84e-4a03-a429-c25bcd89d238/kube-auth-proxy/0.log"
Sep 10 15:17:54.755427 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:54.755401    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59ccbd4756-2q8pl_f9b16477-9d87-43a7-9f53-2f5621126de6/router/0.log"
Sep 10 15:17:55.622980 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:55.622946    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6lmdg_f947de55-ffe1-4f80-920c-25f4212290bc/serve-healthcheck-canary/0.log"
Sep 10 15:17:56.178985 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:56.178957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-z9mrr_2c545044-fe16-41b2-8245-4854e96cc5d3/insights-operator/0.log"
Sep 10 15:17:56.179290 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:56.179274    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-z9mrr_2c545044-fe16-41b2-8245-4854e96cc5d3/insights-operator/1.log"
Sep 10 15:17:56.204471 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:56.204447    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5249m_f03b9612-34b5-4495-b105-777d664603fa/kube-rbac-proxy/0.log"
Sep 10 15:17:56.231813 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:56.231790    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5249m_f03b9612-34b5-4495-b105-777d664603fa/exporter/0.log"
Sep 10 15:17:56.257959 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:56.257928    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5249m_f03b9612-34b5-4495-b105-777d664603fa/extractor/0.log"
Sep 10 15:17:57.976763 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:57.976738    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-prcmw/perf-node-gather-daemonset-8l2gw"
Sep 10 15:17:58.570566 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:58.570538    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q692j_37acfe05-5214-4191-abb7-741d7d0e46b4/manager/1.log"
Sep 10 15:17:58.570733 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:58.570611    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-q692j_37acfe05-5214-4191-abb7-741d7d0e46b4/manager/0.log"
Sep 10 15:17:58.717874 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:58.717844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-ltrn4_f881eda7-2589-4f11-818b-12fbbdc66305/manager/0.log"
Sep 10 15:17:58.767611 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:58.767581    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5bcc5b4d4d-2h97w_071994e7-4084-4a07-b5af-c6426571c6e3/manager/0.log"
Sep 10 15:17:59.049487 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:59.049451    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f84f89d7-mdzcx_b96e3a59-a1d6-4388-b780-7d0b6e4029db/manager/0.log"
Sep 10 15:17:59.068256 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:17:59.068230    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f84f89d7-mdzcx_b96e3a59-a1d6-4388-b780-7d0b6e4029db/copy-manifests/0.log"
Sep 10 15:18:01.151159 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:01.151128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b7fd47cd-9h88c_52d2e320-c84e-4a03-a429-c25bcd89d238/kube-auth-proxy/0.log"
Sep 10 15:18:01.266603 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:01.266484    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59ccbd4756-2q8pl_f9b16477-9d87-43a7-9f53-2f5621126de6/router/0.log"
Sep 10 15:18:02.936902 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:02.936844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79b7fd47cd-9h88c_52d2e320-c84e-4a03-a429-c25bcd89d238/kube-auth-proxy/0.log"
Sep 10 15:18:03.051908 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:03.051866    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59ccbd4756-2q8pl_f9b16477-9d87-43a7-9f53-2f5621126de6/router/0.log"
Sep 10 15:18:06.171827 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:06.171792    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-t5w7p_e3084961-1d48-497e-b105-b9ad92744ee5/kube-storage-version-migrator-operator/1.log"
Sep 10 15:18:06.172918 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:06.172899    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-t5w7p_e3084961-1d48-497e-b105-b9ad92744ee5/kube-storage-version-migrator-operator/0.log"
Sep 10 15:18:07.356870 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.356844    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9dr7_0e12a819-fff9-494a-98f3-6df6aba43abe/kube-multus-additional-cni-plugins/0.log"
Sep 10 15:18:07.379584 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.379561    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9dr7_0e12a819-fff9-494a-98f3-6df6aba43abe/egress-router-binary-copy/0.log"
Sep 10 15:18:07.401205 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.401180    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9dr7_0e12a819-fff9-494a-98f3-6df6aba43abe/cni-plugins/0.log"
Sep 10 15:18:07.423950 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.423929    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9dr7_0e12a819-fff9-494a-98f3-6df6aba43abe/bond-cni-plugin/0.log"
Sep 10 15:18:07.446541 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.446500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9dr7_0e12a819-fff9-494a-98f3-6df6aba43abe/routeoverride-cni/0.log"
Sep 10 15:18:07.468724 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.468700    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9dr7_0e12a819-fff9-494a-98f3-6df6aba43abe/whereabouts-cni-bincopy/0.log"
Sep 10 15:18:07.490437 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.490413    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d9dr7_0e12a819-fff9-494a-98f3-6df6aba43abe/whereabouts-cni/0.log"
Sep 10 15:18:07.786152 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.786126    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vzpmc_5ce228a8-7497-4cb3-9d95-eb279938f9af/kube-multus/0.log"
Sep 10 15:18:07.908725 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.908694    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lk75l_34c67475-95fc-480f-b867-0549df8aff57/network-metrics-daemon/0.log"
Sep 10 15:18:07.928096 ip-10-0-141-186 kubenswrapper[2563]: I0910 15:18:07.928072    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lk75l_34c67475-95fc-480f-b867-0549df8aff57/kube-rbac-proxy/0.log"