Sep 11 15:55:06.159521 ip-10-0-141-101 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 15:55:06.592913 ip-10-0-141-101 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 15:55:06.592913 ip-10-0-141-101 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 15:55:06.592913 ip-10-0-141-101 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 15:55:06.592913 ip-10-0-141-101 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 15:55:06.592913 ip-10-0-141-101 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 15:55:06.595251 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.595160    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 15:55:06.597998 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.597984    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:06.597998 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.597999    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598004    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598008    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598012    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598015    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598018    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598021    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598024    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598027    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598030    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598033    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598037    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598040    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598043    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598045    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598048    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598051    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598053    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598056    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:06.598057 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598059    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598061    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598065    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598068    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598070    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598073    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598075    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598078    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598081    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598083    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598086    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598088    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598091    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598093    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598096    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598098    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598101    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598103    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598106    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598108    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:06.598564 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598111    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598113    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598116    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598118    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598121    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598124    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598126    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598129    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598131    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598134    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598136    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598138    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598141    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598144    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598149    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598152    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598155    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598157    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598160    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598163    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:06.599035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598165    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598168    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598170    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598173    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598175    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598177    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598180    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598182    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598185    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598187    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598190    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598192    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598194    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598197    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598199    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598202    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598204    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598207    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598212    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:06.599522 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598215    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598219    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598221    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598224    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598226    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598230    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598232    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598235    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598250    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598253    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598255    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598258    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598260    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598264    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598268    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598271    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598274    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598277    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598280    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598283    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:06.599965 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598963    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598969    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598971    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598974    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598977    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598980    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598983    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598986    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598988    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598991    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598994    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.598997    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599002    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599005    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599008    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599011    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599013    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599016    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599019    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599021    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:06.600447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599026    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599030    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599035    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599038    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599041    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599044    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599047    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599049    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599052    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599056    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599060    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599063    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599066    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599068    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599071    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599073    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599076    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599079    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599081    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:06.600951 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599085    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599088    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599090    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599093    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599097    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599100    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599103    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599106    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599109    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599111    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599114    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599117    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599119    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599122    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599125    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599127    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599130    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599132    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599135    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599137    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:06.601420 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599140    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599142    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599144    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599147    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599149    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599152    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599154    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599158    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599160    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599163    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599165    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599168    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599170    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599173    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599175    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599178    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599181    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599183    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599186    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599189    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:06.601905 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599192    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599194    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599197    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599199    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599202    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599204    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599207    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599209    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599212    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599214    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599217    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599220    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599222    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599224    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599227    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599229    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599232    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599234    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599251    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:06.602423 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599254    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599602    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599606    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599609    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599611    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599614    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599617    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599619    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599622    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599624    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599627    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599630    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599633    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599635    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599638    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599641    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599644    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599646    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599648    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599651    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:06.602865 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599653    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599656    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599658    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599661    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599663    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599665    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599668    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599670    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599673    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599676    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599680    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599683    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599685    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599688    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599690    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599693    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599695    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599697    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599701    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:06.603348 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599705    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599708    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599711    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599713    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599716    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599719    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599722    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599725    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599729    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599733    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599736    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599739    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599742    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599744    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599747    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599750    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599752    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599755    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599759    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599762    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:06.603863 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599764    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599767    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599769    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599772    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599775    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599777    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599780    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599782    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599785    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599787    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599789    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599792    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599794    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599797    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599799    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599802    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599804    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599807    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599810    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599812    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:06.604355 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599815    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599818    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599821    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599823    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599826    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599828    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599831    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599833    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599835    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599838    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599841    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599844    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599846    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599849    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599851    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599854    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599856    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599859    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599861    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599864    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.599866    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:06.604825 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.599935    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.599965    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.599971    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.599975    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.599984    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.599987    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.599991    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.599995    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.599998    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600001    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600005    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600009    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600012    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600015    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600018    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600021    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600024    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600026    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600030    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600033    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600036    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600041    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600044    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600047    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 15:55:06.605327 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600050    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600053    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600056    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600059    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600062    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600065    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600068    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600071    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600075    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600078    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600081    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600084    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600086    2564 flags.go:64] FLAG: --enable-server="true"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600089    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600093    2564 flags.go:64] FLAG: --event-burst="100"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600096    2564 flags.go:64] FLAG: --event-qps="50"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600099    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600104    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600107    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600110    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600115    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600118    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600121    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600124    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600126    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 15:55:06.605891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600130    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600132    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600135    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600138    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600142    2564 flags.go:64] FLAG: --feature-gates=""
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600146    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600149    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600152    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600155    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600158    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600161    2564 flags.go:64] FLAG: --help="false"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600163    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-141-101.ec2.internal"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600166    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600169    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600172    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600175    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600179    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600182    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600184    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600187    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600190    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600193    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600196    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600199    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 11 15:55:06.606575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600203    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600206    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600209    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600211    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600215    2564 flags.go:64] FLAG: --lock-file=""
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600217    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600220    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600223    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600228    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600231    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600234    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600250    2564 flags.go:64] FLAG: --logging-format="text"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600254    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600257    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600260    2564 flags.go:64] FLAG: --manifest-url=""
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600263    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600267    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600270    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600274    2564 flags.go:64] FLAG: --max-pods="110"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600277    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600280    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600283    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600286    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600289    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600292    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 15:55:06.607149 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600295    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600302    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600305    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600307    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600310    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600313    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600318    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600321    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600326    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600328    2564 flags.go:64] FLAG: --port="10250"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600331    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600334    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08dfd59ddd36964da"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600338    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600341    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600344    2564 flags.go:64] FLAG: --register-node="true"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600346    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600350    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600353    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600355    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600359    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600363    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600366    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600369    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600372    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600374    2564 flags.go:64] FLAG: --runonce="false"
Sep 11 15:55:06.608015 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600377    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600380    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600383    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600386    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600389    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600392    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600395    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600398    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600401    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600403    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600406    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600409    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600412    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600415    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600417    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600424    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600426    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600429    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600433    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600436    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600439    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600442    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600445    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600448    2564 flags.go:64] FLAG: --v="2"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600452    2564 flags.go:64] FLAG: --version="false"
Sep 11 15:55:06.608808 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600456    2564 flags.go:64] FLAG: --vmodule=""
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600460    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600463    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600568    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600573    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600576    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600579    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600582    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600585    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600587    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600590    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600593    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600596    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600598    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600601    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600604    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600606    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600609    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600611    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600614    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:06.609437 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600617    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600619    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600624    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600628    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600631    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600634    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600638    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600641    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600644    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600647    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600650    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600653    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600655    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600658    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600661    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600664    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600666    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600669    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600671    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600674    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:06.609908 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600676    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600679    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600681    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600684    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600687    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600689    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600692    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600694    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600697    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600699    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600702    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600704    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600707    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600710    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600713    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600716    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600718    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600721    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600724    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:06.610447 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600727    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600729    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600732    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600735    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600737    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600740    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600742    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600745    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600747    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600750    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600752    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600755    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600758    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600760    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600762    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600765    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600769    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600772    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600775    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600778    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:06.610899 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600781    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600783    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600786    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600789    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600791    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600794    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600796    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600800    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600803    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600806    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600808    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600811    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600813    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600816    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600819    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600822    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600832    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600834    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600837    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600840    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:06.611482 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600842    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600845    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.600847    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.600853    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.609070    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.609084    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.609858    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.609867    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609924    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609929    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609932    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609935    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609938    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609942    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:06.611957 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609945    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609949    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609954    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609957    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609960    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609962    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609966    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609969    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609971    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609974    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609977    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609980    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609983    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609985    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609988    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609991    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609994    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609997    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.609999    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610002    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:06.612308 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610005    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610007    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610010    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610014    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610016    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610019    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610022    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610025    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610027    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610030    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610033    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610035    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610038    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610040    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610043    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610045    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610048    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610050    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610053    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610055    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:06.612783 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610059    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610061    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610064    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610066    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610069    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610071    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610074    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610076    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610079    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610081    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610084    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610086    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610089    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610091    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610095    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610099    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610102    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610105    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610107    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:06.613274 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610110    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610113    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610115    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610118    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610121    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610123    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610126    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610129    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610131    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610134    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610136    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610139    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610142    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610144    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610147    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610150    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610152    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610154    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610157    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610159    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:06.613763 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610162    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610164    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610167    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610169    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610172    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610174    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610176    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610179    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610182    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610186    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610189    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610192    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610195    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610198    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:06.614325 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.610203    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610336    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610343    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610346    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610349    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610352    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610355    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610358    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610361    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610363    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610366    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610368    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610371    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610375    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610378    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610380    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610383    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610385    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610388    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 15:55:06.614673 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610390    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610393    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610395    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610398    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610401    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610403    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610405    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610408    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610411    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610414    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610416    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610419    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610421    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610424    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610426    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610429    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610432    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610434    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610437    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610439    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 15:55:06.615111 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610442    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610446    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610450    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610454    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610457    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610461    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610464    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610467    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610470    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610473    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610476    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610479    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610482    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610484    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610487    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610489    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610492    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610494    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610497    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 15:55:06.615595 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610500    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610503    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610506    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610508    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610511    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610513    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610516    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610519    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610521    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610524    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610526    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610528    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610531    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610534    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610536    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610538    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610541    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610544    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610546    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610549    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 15:55:06.616043 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610551    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610554    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610557    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610560    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610562    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610565    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610567    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610570    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610572    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610575    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610578    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610580    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610583    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610586    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610588    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610591    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610593    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610596    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610598    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610601    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 15:55:06.616523 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610603    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 15:55:06.616979 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:06.610605    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 15:55:06.616979 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.610610    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 15:55:06.616979 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.610714    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 15:55:06.616979 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.613843    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 15:55:06.616979 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.614857    2564 server.go:1019] "Starting client certificate rotation"
Sep 11 15:55:06.616979 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.614951    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 15:55:06.616979 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.614983    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 15:55:06.617492 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.617479    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 15:55:06.618630 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.618612    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 15:55:06.627232 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.627212    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 11 15:55:06.630104 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.630091    2564 log.go:25] "Validated CRI v1 image API"
Sep 11 15:55:06.633005 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.632990    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 15:55:06.636924 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.636903    2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bdf24cd6-cd8b-4e57-9cac-7aa6c7824e46:/dev/nvme0n1p4 cdc200e6-9bb2-4b29-97f8-9e489ea0c08b:/dev/nvme0n1p3]
Sep 11 15:55:06.637011 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.636922    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 15:55:06.641805 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.641684    2564 manager.go:216] Machine: {Timestamp:2026-09-11 15:55:06.640361046 +0000 UTC m=+0.358336296 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200675 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aab9d7664e10170b6241341b376f2 SystemUUID:ec2aab9d-7664-e101-70b6-241341b376f2 BootID:b3b63822-e5bf-473f-a072-872dce51ca03 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e0:0e:f0:d5:93 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e0:0e:f0:d5:93 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fe:9e:08:c5:c6:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 15:55:06.641805 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.641794    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 15:55:06.641970 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.641895    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 15:55:06.642987 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.642970    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 15:55:06.645357 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.645321    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 15:55:06.645483 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.645361    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 15:55:06.646129 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.646120    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 15:55:06.646164 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.646132    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 15:55:06.646164 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.646139    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 15:55:06.646682 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.646671    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 15:55:06.647465 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.647455    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 15:55:06.650190 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.650179    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 15:55:06.650316 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.650307    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 15:55:06.650355 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.650337    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 15:55:06.650355 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.650350    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 15:55:06.650407 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.650368    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 15:55:06.650407 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.650381    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 11 15:55:06.650407 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.650392    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 15:55:06.650751 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.650739    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 15:55:06.650810 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.650757    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 15:55:06.653812 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.653781    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 15:55:06.655968 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.655954    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 15:55:06.656014 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.655985    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 15:55:06.657763 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657753    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 15:55:06.657803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657768    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 15:55:06.657803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657773    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 15:55:06.657803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657778    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 15:55:06.657803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657786    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 15:55:06.657803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657792    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 15:55:06.657803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657805    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 15:55:06.657803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657811    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 15:55:06.657994 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657821    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 15:55:06.657994 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657826    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 15:55:06.657994 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657834    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 15:55:06.657994 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.657839    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 15:55:06.658543 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.658533    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 15:55:06.658581 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.658546    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 15:55:06.662265 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.662231    2564 server.go:1268] "Started kubelet"
Sep 11 15:55:06.662390 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.662343    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 15:55:06.662464 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.662406    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 15:55:06.662464 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.662434    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 15:55:06.664353 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.664334    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 15:55:06.665384 ip-10-0-141-101 systemd[1]: Started Kubernetes Kubelet.
Sep 11 15:55:06.666396 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.665618    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 15:55:06.667688 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.667670    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 15:55:06.669408 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.669394    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 15:55:06.670169 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.670150    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 15:55:06.670598 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.670582    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:06.670713 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.670682    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 15:55:06.670713 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.670701    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 15:55:06.670846 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.670833    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 15:55:06.670981 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.670964    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 15:55:06.670981 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.670981    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 15:55:06.671406 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.671375    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 15:55:06.671508 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.671422    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 15:55:06.671508 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.671437    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 15:55:06.672010 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.671992    2564 factory.go:55] Registering systemd factory
Sep 11 15:55:06.672099 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.672036    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 11 15:55:06.672291 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.672275    2564 factory.go:153] Registering CRI-O factory
Sep 11 15:55:06.672291 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.672292    2564 factory.go:223] Registration of the crio container factory successfully
Sep 11 15:55:06.672405 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.672338    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 15:55:06.672405 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.672355    2564 factory.go:103] Registering Raw factory
Sep 11 15:55:06.672405 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.672368    2564 manager.go:1196] Started watching for new ooms in manager
Sep 11 15:55:06.672861 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.671788    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-101.ec2.internal.18d44f44484b68a2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-101.ec2.internal,UID:ip-10-0-141-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-101.ec2.internal,},FirstTimestamp:2026-09-11 15:55:06.662201506 +0000 UTC m=+0.380176620,LastTimestamp:2026-09-11 15:55:06.662201506 +0000 UTC m=+0.380176620,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-101.ec2.internal,}"
Sep 11 15:55:06.673181 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.673169    2564 manager.go:349] Starting recovery of all containers
Sep 11 15:55:06.678837 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.678810    2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 15:55:06.680126 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.680098    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 15:55:06.681584 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.681563    2564 manager.go:354] Recovery completed
Sep 11 15:55:06.686947 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.686930    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:06.688522 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.688505    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:06.688602 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.688532    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:06.688602 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.688541    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:06.689014 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.688999    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 15:55:06.689014 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.689013    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 15:55:06.689116 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.689027    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 15:55:06.689116 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.689004    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:06.689666 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.689651    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:06.689730 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.689676    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:06.689730 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.689687    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:06.690314 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.690298    2564 policy_none.go:49] "None policy: Start"
Sep 11 15:55:06.690314 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.690314    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 15:55:06.690441 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.690324    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 15:55:06.690825 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.690734    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-101.ec2.internal.18d44f4449dcfec2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-101.ec2.internal,UID:ip-10-0-141-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-101.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-101.ec2.internal,},FirstTimestamp:2026-09-11 15:55:06.688519874 +0000 UTC m=+0.406494985,LastTimestamp:2026-09-11 15:55:06.688519874 +0000 UTC m=+0.406494985,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-101.ec2.internal,}"
Sep 11 15:55:06.691187 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.691176    2564 policy_none.go:47] "Start"
Sep 11 15:55:06.704434 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.704314    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s556n"
Sep 11 15:55:06.706312 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.706209    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-101.ec2.internal.18d44f4449dd3e59  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-101.ec2.internal,UID:ip-10-0-141-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-101.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-101.ec2.internal,},FirstTimestamp:2026-09-11 15:55:06.688536153 +0000 UTC m=+0.406511264,LastTimestamp:2026-09-11 15:55:06.688536153 +0000 UTC m=+0.406511264,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-101.ec2.internal,}"
Sep 11 15:55:06.716790 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.716768    2564 manager.go:337] "Starting Device Plugin manager"
Sep 11 15:55:06.716891 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.716813    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 15:55:06.716891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.716827    2564 server.go:87] "Starting device plugin registration server"
Sep 11 15:55:06.717102 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.717088    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 15:55:06.717180 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.717103    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 15:55:06.717270 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.717257    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 15:55:06.717845 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.717323    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 15:55:06.717845 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.717330    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 15:55:06.717845 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.717778    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 15:55:06.717845 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.717819    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:06.718187 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.718174    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s556n"
Sep 11 15:55:06.803150 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.803122    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 15:55:06.804281 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.804267    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 15:55:06.804363 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.804307    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 15:55:06.804363 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.804334    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 15:55:06.804450 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.804403    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 15:55:06.807643 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.807628    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:06.817414 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.817400    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:06.818171 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.818154    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:06.818264 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.818182    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:06.818264 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.818197    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:06.818264 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.818222    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-101.ec2.internal"
Sep 11 15:55:06.838132 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.838111    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-101.ec2.internal"
Sep 11 15:55:06.838132 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.838133    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-101.ec2.internal\": node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:06.904756 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.904691    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"]
Sep 11 15:55:06.904756 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.904752    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:06.905493 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.905479    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:06.905559 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.905507    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:06.905559 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.905520    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:06.906825 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.906814    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:06.906932 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.906920    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:06.906969 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.906944    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:06.907486 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.907467    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:06.907564 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.907491    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:06.907564 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.907500    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:06.908144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.908127    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:06.908218 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.908159    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:06.908218 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.908174    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:06.909599 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.909584    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:06.909687 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.909607    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:06.910178 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.910165    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:06.910270 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.910186    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:06.910270 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.910196    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:06.911183 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.911161    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:06.923738 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.923722    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-101.ec2.internal\" not found" node="ip-10-0-141-101.ec2.internal"
Sep 11 15:55:06.927514 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:06.927501    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-101.ec2.internal\" not found" node="ip-10-0-141-101.ec2.internal"
Sep 11 15:55:06.972300 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.972285    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8494a3e6a9c170896d58705ab106f0d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-101.ec2.internal\" (UID: \"e8494a3e6a9c170896d58705ab106f0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:06.972381 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.972306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6d8b968f918649697ed3c7789abede-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal\" (UID: \"9d6d8b968f918649697ed3c7789abede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:06.972381 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:06.972322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6d8b968f918649697ed3c7789abede-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal\" (UID: \"9d6d8b968f918649697ed3c7789abede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.011687 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:07.011670    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:07.073312 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.073296    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8494a3e6a9c170896d58705ab106f0d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-101.ec2.internal\" (UID: \"e8494a3e6a9c170896d58705ab106f0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.073366 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.073318    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6d8b968f918649697ed3c7789abede-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal\" (UID: \"9d6d8b968f918649697ed3c7789abede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.073366 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.073332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6d8b968f918649697ed3c7789abede-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal\" (UID: \"9d6d8b968f918649697ed3c7789abede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.073366 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.073357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d6d8b968f918649697ed3c7789abede-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal\" (UID: \"9d6d8b968f918649697ed3c7789abede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.073468 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.073382    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8494a3e6a9c170896d58705ab106f0d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-101.ec2.internal\" (UID: \"e8494a3e6a9c170896d58705ab106f0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.073468 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.073417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d6d8b968f918649697ed3c7789abede-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal\" (UID: \"9d6d8b968f918649697ed3c7789abede\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.112547 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:07.112531    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:07.213130 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:07.213084    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:07.226380 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.226368    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.231053 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.231037    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.314109 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:07.314081    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:07.415722 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:07.415691    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:07.518719 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:07.518656    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:07.543690 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.543669    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:07.614890 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.614864    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 15:55:07.615383 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.615001    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 15:55:07.615383 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.615044    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 15:55:07.619464 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:07.619446    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:07.642963 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.642942    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:07.720194 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:07.720141    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:07.722395 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.722367    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 15:50:06 +0000 UTC" deadline="2028-03-24 07:14:31.722769921 +0000 UTC"
Sep 11 15:55:07.722451 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.722396    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13431h19m24.000377005s"
Sep 11 15:55:07.758551 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.758527    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 15:55:07.759686 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.759671    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 15:55:07.759747 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.759701    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 15:55:07.759747 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.759712    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeHasSufficientPID"
Sep 11 15:55:07.799218 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:07.799182    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6d8b968f918649697ed3c7789abede.slice/crio-645c021e703f847a3cafdd8d4c1111c5582d36d9203849c81a326c70e8ea4891 WatchSource:0}: Error finding container 645c021e703f847a3cafdd8d4c1111c5582d36d9203849c81a326c70e8ea4891: Status 404 returned error can't find the container with id 645c021e703f847a3cafdd8d4c1111c5582d36d9203849c81a326c70e8ea4891
Sep 11 15:55:07.800287 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:07.800262    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8494a3e6a9c170896d58705ab106f0d.slice/crio-5dc41b4c030c7e448aa9515ce43ce03893c58144112583684a4c26a8db4a6def WatchSource:0}: Error finding container 5dc41b4c030c7e448aa9515ce43ce03893c58144112583684a4c26a8db4a6def: Status 404 returned error can't find the container with id 5dc41b4c030c7e448aa9515ce43ce03893c58144112583684a4c26a8db4a6def
Sep 11 15:55:07.804338 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.804319    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 15:55:07.807664 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.807631    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal" event={"ID":"9d6d8b968f918649697ed3c7789abede","Type":"ContainerStarted","Data":"645c021e703f847a3cafdd8d4c1111c5582d36d9203849c81a326c70e8ea4891"}
Sep 11 15:55:07.808993 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.808973    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal" event={"ID":"e8494a3e6a9c170896d58705ab106f0d","Type":"ContainerStarted","Data":"5dc41b4c030c7e448aa9515ce43ce03893c58144112583684a4c26a8db4a6def"}
Sep 11 15:55:07.821310 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:07.821293    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-101.ec2.internal\" not found"
Sep 11 15:55:07.845541 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.845525    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:07.870738 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.870714    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.886083 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.886064    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 15:55:07.887145 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.887130    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal"
Sep 11 15:55:07.896372 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:07.896350    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 15:55:08.652193 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.652162    2564 apiserver.go:52] "Watching apiserver"
Sep 11 15:55:08.661451 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.661427    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 15:55:08.663892 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.663857    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-4dp94","kube-system/global-pull-secret-syncer-74rvc","kube-system/konnectivity-agent-2nz4w","openshift-cluster-node-tuning-operator/tuned-ntxnc","openshift-image-registry/node-ca-cwf6l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal","openshift-multus/multus-9j4bk","openshift-multus/multus-additional-cni-plugins-stj8f","openshift-multus/network-metrics-daemon-cfrmq","kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6","openshift-dns/node-resolver-9m89b","openshift-network-diagnostics/network-check-target-k5h9f","openshift-network-operator/iptables-alerter-w5fdn"]
Sep 11 15:55:08.665501 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.665473    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.666581 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.666559    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:08.666687 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.666629    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:08.667652 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.667629    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:08.668615 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.668559    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 15:55:08.668615 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.668577    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fpwg7\""
Sep 11 15:55:08.668773 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.668627    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 15:55:08.668960 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.668933    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 15:55:08.669052 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.668946    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 15:55:08.669221 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.669203    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.670339 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.670302    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7rnp\""
Sep 11 15:55:08.672076 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.670310    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 15:55:08.672076 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.670398    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 15:55:08.672076 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.670420    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 15:55:08.672076 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.671292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.672076 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.671913    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 15:55:08.672076 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.671921    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 15:55:08.672076 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.672001    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h56bx\""
Sep 11 15:55:08.673368 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.673347    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.674141 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.674120    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 15:55:08.674270 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.674233    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 15:55:08.674370 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.674355    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 15:55:08.675071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.674642    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xppsj\""
Sep 11 15:55:08.675988 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.675969    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:08.676079 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.676036    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:08.676141 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.676108    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m5lk7\""
Sep 11 15:55:08.676379 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.676357    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 15:55:08.676439 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.676423    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 15:55:08.676642 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.676623    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 15:55:08.677545 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.677527    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.679341 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.679321    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:08.679435 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.679389    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:08.680397 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.680377    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 15:55:08.680503 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.680485    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 15:55:08.680601 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.680582    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 15:55:08.681067 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.681049    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 15:55:08.681499 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.681478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.682803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682572    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-h99f2\""
Sep 11 15:55:08.682803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682655    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96b4ef1a-6e79-42e2-8f38-6b7355fde851-iptables-alerter-script\") pod \"iptables-alerter-w5fdn\" (UID: \"96b4ef1a-6e79-42e2-8f38-6b7355fde851\") " pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.682803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682686    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-var-lib-cni-bin\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.682803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-var-lib-kubelet\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.682803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682751    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f7e01ee-e3d6-48c6-b567-ad87583a88bc-konnectivity-ca\") pod \"konnectivity-agent-2nz4w\" (UID: \"8f7e01ee-e3d6-48c6-b567-ad87583a88bc\") " pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:08.682803 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-sysconfig\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-systemd\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682868    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdk6\" (UniqueName: \"kubernetes.io/projected/96b4ef1a-6e79-42e2-8f38-6b7355fde851-kube-api-access-jgdk6\") pod \"iptables-alerter-w5fdn\" (UID: \"96b4ef1a-6e79-42e2-8f38-6b7355fde851\") " pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682904    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2199b2b-e15f-469b-825a-05423d6fa5ff-cni-binary-copy\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-conf-dir\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682951    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-host\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.682976    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czrrj\" (UniqueName: \"kubernetes.io/projected/c7165280-6c2a-4719-8420-60faeeac4f82-kube-api-access-czrrj\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683013    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-socket-dir-parent\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683038    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-run-k8s-cni-cncf-io\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683061    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-run-netns\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683088    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-run-multus-certs\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-etc-kubernetes\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683144 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-system-cni-dir\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-cni-dir\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683202    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-cnibin\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683228    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb6mx\" (UniqueName: \"kubernetes.io/projected/b2199b2b-e15f-469b-825a-05423d6fa5ff-kube-api-access-zb6mx\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683281    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f7e01ee-e3d6-48c6-b567-ad87583a88bc-agent-certs\") pod \"konnectivity-agent-2nz4w\" (UID: \"8f7e01ee-e3d6-48c6-b567-ad87583a88bc\") " pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683285    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683332    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bf6c33a-d45b-40b1-a1ba-f42381cd72ad-host\") pod \"node-ca-cwf6l\" (UID: \"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad\") " pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683382    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bf6c33a-d45b-40b1-a1ba-f42381cd72ad-serviceca\") pod \"node-ca-cwf6l\" (UID: \"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad\") " pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683407    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96b4ef1a-6e79-42e2-8f38-6b7355fde851-host-slash\") pod \"iptables-alerter-w5fdn\" (UID: \"96b4ef1a-6e79-42e2-8f38-6b7355fde851\") " pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683434    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-hostroot\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683502    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-daemon-config\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683534    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-sysctl-d\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683619    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683650    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-sysctl-conf\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-run\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683735    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.683745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683749    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-var-lib-cni-multus\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.684537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683799    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-kubernetes\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.684537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683827    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-var-lib-kubelet\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.684537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683856    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7165280-6c2a-4719-8420-60faeeac4f82-etc-tuned\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.684537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683872    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 15:55:08.684537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7165280-6c2a-4719-8420-60faeeac4f82-tmp\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.684537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683906    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7d77\" (UniqueName: \"kubernetes.io/projected/1bf6c33a-d45b-40b1-a1ba-f42381cd72ad-kube-api-access-r7d77\") pod \"node-ca-cwf6l\" (UID: \"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad\") " pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.684537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-modprobe-d\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.684537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.683987    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-sys\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.684537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.684009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-lib-modules\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.685038 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.684694    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.685568 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.685301    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b549p\""
Sep 11 15:55:08.686917 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.686516    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 15:55:08.687332 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.687097    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 15:55:08.690363 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.689545    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pssvp\""
Sep 11 15:55:08.690363 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.689672    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 15:55:08.690363 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.689844    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 15:55:08.690363 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.689987    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rl7kn\""
Sep 11 15:55:08.690363 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.690157    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 15:55:08.690363 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.690236    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 15:55:08.690680 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.690585    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 15:55:08.710567 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.710542    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zpxq7"
Sep 11 15:55:08.718757 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.718736    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zpxq7"
Sep 11 15:55:08.771960 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.771939    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 15:55:08.784727 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784699    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.784850 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784740    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-system-cni-dir\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.784850 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784770    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdmvg\" (UniqueName: \"kubernetes.io/projected/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-kube-api-access-zdmvg\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.784850 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-systemd\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.784850 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784826    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-slash\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.785071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-host\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.785071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-run-k8s-cni-cncf-io\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.785071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-run-multus-certs\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.785071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784924    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.785071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-socket-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.785071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.784985    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.785071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-var-lib-openvswitch\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.785071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785033    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.785071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785058    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-cni-dir\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785103    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-host\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785105    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-sys-fs\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785108    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-run-k8s-cni-cncf-io\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8be5b91-0905-4d4a-83b7-4e80f4b3545e-tmp-dir\") pod \"node-resolver-9m89b\" (UID: \"b8be5b91-0905-4d4a-83b7-4e80f4b3545e\") " pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785175    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-systemd\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785178    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-run-multus-certs\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785209    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn59r\" (UniqueName: \"kubernetes.io/projected/b8be5b91-0905-4d4a-83b7-4e80f4b3545e-kube-api-access-bn59r\") pod \"node-resolver-9m89b\" (UID: \"b8be5b91-0905-4d4a-83b7-4e80f4b3545e\") " pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785237    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-etc-openvswitch\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785309    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-cni-netd\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785322    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-cni-dir\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785333    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16098cd9-fcba-4c67-bba4-bce8820024a2-ovnkube-config\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785366    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785396    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb6mx\" (UniqueName: \"kubernetes.io/projected/b2199b2b-e15f-469b-825a-05423d6fa5ff-kube-api-access-zb6mx\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-systemd-units\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-cnibin\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.785610 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f7e01ee-e3d6-48c6-b567-ad87583a88bc-agent-certs\") pod \"konnectivity-agent-2nz4w\" (UID: \"8f7e01ee-e3d6-48c6-b567-ad87583a88bc\") " pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785570    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bf6c33a-d45b-40b1-a1ba-f42381cd72ad-host\") pod \"node-ca-cwf6l\" (UID: \"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad\") " pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bf6c33a-d45b-40b1-a1ba-f42381cd72ad-serviceca\") pod \"node-ca-cwf6l\" (UID: \"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad\") " pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96b4ef1a-6e79-42e2-8f38-6b7355fde851-host-slash\") pod \"iptables-alerter-w5fdn\" (UID: \"96b4ef1a-6e79-42e2-8f38-6b7355fde851\") " pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16098cd9-fcba-4c67-bba4-bce8820024a2-ovn-node-metrics-cert\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785654    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-var-lib-cni-multus\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785684    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-registration-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-kubernetes\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785721    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7d77\" (UniqueName: \"kubernetes.io/projected/1bf6c33a-d45b-40b1-a1ba-f42381cd72ad-kube-api-access-r7d77\") pod \"node-ca-cwf6l\" (UID: \"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad\") " pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-run-netns\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785756    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-node-log\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785784    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-modprobe-d\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785785    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-var-lib-cni-multus\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96b4ef1a-6e79-42e2-8f38-6b7355fde851-iptables-alerter-script\") pod \"iptables-alerter-w5fdn\" (UID: \"96b4ef1a-6e79-42e2-8f38-6b7355fde851\") " pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785825    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-var-lib-kubelet\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785869    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7mqw\" (UniqueName: \"kubernetes.io/projected/38668909-95f5-4bbb-84df-f71f4856578a-kube-api-access-k7mqw\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:08.786489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785876    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-kubernetes\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785896    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-run-systemd\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785928    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96b4ef1a-6e79-42e2-8f38-6b7355fde851-host-slash\") pod \"iptables-alerter-w5fdn\" (UID: \"96b4ef1a-6e79-42e2-8f38-6b7355fde851\") " pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.785970    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f7e01ee-e3d6-48c6-b567-ad87583a88bc-konnectivity-ca\") pod \"konnectivity-agent-2nz4w\" (UID: \"8f7e01ee-e3d6-48c6-b567-ad87583a88bc\") " pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786001    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-sysconfig\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786005    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-var-lib-kubelet\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786039    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1bf6c33a-d45b-40b1-a1ba-f42381cd72ad-host\") pod \"node-ca-cwf6l\" (UID: \"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad\") " pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786043    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-modprobe-d\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdk6\" (UniqueName: \"kubernetes.io/projected/96b4ef1a-6e79-42e2-8f38-6b7355fde851-kube-api-access-jgdk6\") pod \"iptables-alerter-w5fdn\" (UID: \"96b4ef1a-6e79-42e2-8f38-6b7355fde851\") " pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786089    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-sysconfig\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786126    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2199b2b-e15f-469b-825a-05423d6fa5ff-cni-binary-copy\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1bf6c33a-d45b-40b1-a1ba-f42381cd72ad-serviceca\") pod \"node-ca-cwf6l\" (UID: \"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad\") " pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786164    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-conf-dir\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786267    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrdr\" (UniqueName: \"kubernetes.io/projected/682c9904-4b50-46ad-8429-250c436874e6-kube-api-access-8zrdr\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-conf-dir\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786329    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:08.787311 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czrrj\" (UniqueName: \"kubernetes.io/projected/c7165280-6c2a-4719-8420-60faeeac4f82-kube-api-access-czrrj\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786415    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-socket-dir-parent\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786433    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-run-netns\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786547    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-run-netns\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786591    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96b4ef1a-6e79-42e2-8f38-6b7355fde851-iptables-alerter-script\") pod \"iptables-alerter-w5fdn\" (UID: \"96b4ef1a-6e79-42e2-8f38-6b7355fde851\") " pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786606    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-socket-dir-parent\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-etc-kubernetes\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-etc-kubernetes\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-kubelet\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786683    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8f7e01ee-e3d6-48c6-b567-ad87583a88bc-konnectivity-ca\") pod \"konnectivity-agent-2nz4w\" (UID: \"8f7e01ee-e3d6-48c6-b567-ad87583a88bc\") " pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786725    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-run-openvswitch\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786753    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-run-ovn\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786769    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2199b2b-e15f-469b-825a-05423d6fa5ff-cni-binary-copy\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786777    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16098cd9-fcba-4c67-bba4-bce8820024a2-ovnkube-script-lib\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786804    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-system-cni-dir\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-cnibin\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786868    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lrg\" (UniqueName: \"kubernetes.io/projected/16098cd9-fcba-4c67-bba4-bce8820024a2-kube-api-access-k2lrg\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.788206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786904    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-device-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-system-cni-dir\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-cnibin\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-cni-bin\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16098cd9-fcba-4c67-bba4-bce8820024a2-env-overrides\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.786994    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8844b9e8-33aa-4253-8570-bf7b1b1c346e-kubelet-config\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787010    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-hostroot\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-daemon-config\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787075    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-log-socket\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787092    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-sysctl-d\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-sysctl-conf\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787154    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-run\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787195    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-cni-binary-copy\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787236    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-var-lib-kubelet\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787261    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7165280-6c2a-4719-8420-60faeeac4f82-etc-tuned\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787270    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-sysctl-d\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789112 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7165280-6c2a-4719-8420-60faeeac4f82-tmp\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787311    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-etc-sysctl-conf\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787323    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-hostroot\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787350    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8be5b91-0905-4d4a-83b7-4e80f4b3545e-hosts-file\") pod \"node-resolver-9m89b\" (UID: \"b8be5b91-0905-4d4a-83b7-4e80f4b3545e\") " pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8844b9e8-33aa-4253-8570-bf7b1b1c346e-dbus\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787387    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-var-lib-kubelet\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-sys\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787391    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-run\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-lib-modules\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787458    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-var-lib-cni-bin\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787545    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2199b2b-e15f-469b-825a-05423d6fa5ff-host-var-lib-cni-bin\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787432    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-sys\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787599    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2199b2b-e15f-469b-825a-05423d6fa5ff-multus-daemon-config\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.789982 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.787649    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7165280-6c2a-4719-8420-60faeeac4f82-lib-modules\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.790696 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.790666    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7165280-6c2a-4719-8420-60faeeac4f82-etc-tuned\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.790761 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.790708    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7165280-6c2a-4719-8420-60faeeac4f82-tmp\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.791072 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.791049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8f7e01ee-e3d6-48c6-b567-ad87583a88bc-agent-certs\") pod \"konnectivity-agent-2nz4w\" (UID: \"8f7e01ee-e3d6-48c6-b567-ad87583a88bc\") " pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:08.793671 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.793635    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:08.793671 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.793672    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:08.794351 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.793686    2564 projected.go:196] Error preparing data for projected volume kube-api-access-585lg for pod openshift-network-diagnostics/network-check-target-k5h9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:08.794351 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.793827    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg podName:c0183175-4d09-407f-b523-00a608be58b8 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:09.293806041 +0000 UTC m=+3.011781162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-585lg" (UniqueName: "kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg") pod "network-check-target-k5h9f" (UID: "c0183175-4d09-407f-b523-00a608be58b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:08.796088 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.796062    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb6mx\" (UniqueName: \"kubernetes.io/projected/b2199b2b-e15f-469b-825a-05423d6fa5ff-kube-api-access-zb6mx\") pod \"multus-9j4bk\" (UID: \"b2199b2b-e15f-469b-825a-05423d6fa5ff\") " pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.796698 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.796534    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdk6\" (UniqueName: \"kubernetes.io/projected/96b4ef1a-6e79-42e2-8f38-6b7355fde851-kube-api-access-jgdk6\") pod \"iptables-alerter-w5fdn\" (UID: \"96b4ef1a-6e79-42e2-8f38-6b7355fde851\") " pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:08.797568 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.797522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7d77\" (UniqueName: \"kubernetes.io/projected/1bf6c33a-d45b-40b1-a1ba-f42381cd72ad-kube-api-access-r7d77\") pod \"node-ca-cwf6l\" (UID: \"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad\") " pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:08.797923 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.797897    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czrrj\" (UniqueName: \"kubernetes.io/projected/c7165280-6c2a-4719-8420-60faeeac4f82-kube-api-access-czrrj\") pod \"tuned-ntxnc\" (UID: \"c7165280-6c2a-4719-8420-60faeeac4f82\") " pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:08.845125 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.845104    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:08.887731 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-system-cni-dir\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.887862 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887741    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdmvg\" (UniqueName: \"kubernetes.io/projected/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-kube-api-access-zdmvg\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.887862 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-slash\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.887862 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.887862 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887804    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-system-cni-dir\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.887862 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887820    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-socket-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.887862 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887868    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-var-lib-openvswitch\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887873    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-slash\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887894    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887934    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-sys-fs\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-var-lib-openvswitch\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8be5b91-0905-4d4a-83b7-4e80f4b3545e-tmp-dir\") pod \"node-resolver-9m89b\" (UID: \"b8be5b91-0905-4d4a-83b7-4e80f4b3545e\") " pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn59r\" (UniqueName: \"kubernetes.io/projected/b8be5b91-0905-4d4a-83b7-4e80f4b3545e-kube-api-access-bn59r\") pod \"node-resolver-9m89b\" (UID: \"b8be5b91-0905-4d4a-83b7-4e80f4b3545e\") " pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.887983    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-socket-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-sys-fs\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888016    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-etc-openvswitch\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888040    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-cni-netd\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16098cd9-fcba-4c67-bba4-bce8820024a2-ovnkube-config\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888087    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:08.888130 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-systemd-units\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888130    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-cni-netd\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888136    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-cnibin\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888162    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16098cd9-fcba-4c67-bba4-bce8820024a2-ovn-node-metrics-cert\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-systemd-units\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.888218    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888239    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-registration-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888262    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-cnibin\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888288    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-run-netns\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888298    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8be5b91-0905-4d4a-83b7-4e80f4b3545e-tmp-dir\") pod \"node-resolver-9m89b\" (UID: \"b8be5b91-0905-4d4a-83b7-4e80f4b3545e\") " pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.888310    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret podName:8844b9e8-33aa-4253-8570-bf7b1b1c346e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:09.388290914 +0000 UTC m=+3.106266034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret") pod "global-pull-secret-syncer-74rvc" (UID: "8844b9e8-33aa-4253-8570-bf7b1b1c346e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888160    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-etc-openvswitch\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-run-netns\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-registration-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888373    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-node-log\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-node-log\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.888910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888413    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mqw\" (UniqueName: \"kubernetes.io/projected/38668909-95f5-4bbb-84df-f71f4856578a-kube-api-access-k7mqw\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-run-systemd\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888498    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-run-systemd\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888496    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrdr\" (UniqueName: \"kubernetes.io/projected/682c9904-4b50-46ad-8429-250c436874e6-kube-api-access-8zrdr\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888565    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-kubelet\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888610    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-run-openvswitch\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888633    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-run-ovn\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.888639    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888643    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-kubelet\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888656    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16098cd9-fcba-4c67-bba4-bce8820024a2-ovnkube-script-lib\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888687    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lrg\" (UniqueName: \"kubernetes.io/projected/16098cd9-fcba-4c67-bba4-bce8820024a2-kube-api-access-k2lrg\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888694    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-run-ovn\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:08.888710    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs podName:38668909-95f5-4bbb-84df-f71f4856578a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:09.388691118 +0000 UTC m=+3.106666233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs") pod "network-metrics-daemon-cfrmq" (UID: "38668909-95f5-4bbb-84df-f71f4856578a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888737    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888737    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-run-openvswitch\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.889726 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888765    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-device-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-cni-bin\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888823    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16098cd9-fcba-4c67-bba4-bce8820024a2-env-overrides\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8844b9e8-33aa-4253-8570-bf7b1b1c346e-kubelet-config\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888874    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/16098cd9-fcba-4c67-bba4-bce8820024a2-ovnkube-config\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/682c9904-4b50-46ad-8429-250c436874e6-device-dir\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8844b9e8-33aa-4253-8570-bf7b1b1c346e-kubelet-config\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888879    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-cni-bin\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-log-socket\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888980    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-log-socket\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.888978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-cni-binary-copy\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889021    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8be5b91-0905-4d4a-83b7-4e80f4b3545e-hosts-file\") pod \"node-resolver-9m89b\" (UID: \"b8be5b91-0905-4d4a-83b7-4e80f4b3545e\") " pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8844b9e8-33aa-4253-8570-bf7b1b1c346e-dbus\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889072    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b8be5b91-0905-4d4a-83b7-4e80f4b3545e-hosts-file\") pod \"node-resolver-9m89b\" (UID: \"b8be5b91-0905-4d4a-83b7-4e80f4b3545e\") " pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/16098cd9-fcba-4c67-bba4-bce8820024a2-host-run-ovn-kubernetes\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.890418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8844b9e8-33aa-4253-8570-bf7b1b1c346e-dbus\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:08.891203 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889197    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/16098cd9-fcba-4c67-bba4-bce8820024a2-env-overrides\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.891203 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889201    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/16098cd9-fcba-4c67-bba4-bce8820024a2-ovnkube-script-lib\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.891203 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889383    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.891203 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-cni-binary-copy\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.891203 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.889575    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.891203 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.890916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/16098cd9-fcba-4c67-bba4-bce8820024a2-ovn-node-metrics-cert\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.902301 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.902230    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn59r\" (UniqueName: \"kubernetes.io/projected/b8be5b91-0905-4d4a-83b7-4e80f4b3545e-kube-api-access-bn59r\") pod \"node-resolver-9m89b\" (UID: \"b8be5b91-0905-4d4a-83b7-4e80f4b3545e\") " pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:08.904930 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.904904    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrdr\" (UniqueName: \"kubernetes.io/projected/682c9904-4b50-46ad-8429-250c436874e6-kube-api-access-8zrdr\") pod \"aws-ebs-csi-driver-node-c8rs6\" (UID: \"682c9904-4b50-46ad-8429-250c436874e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:08.905424 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.905401    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdmvg\" (UniqueName: \"kubernetes.io/projected/86c3d20c-209e-4ae8-84d8-5fe3542a4fd1-kube-api-access-zdmvg\") pod \"multus-additional-cni-plugins-stj8f\" (UID: \"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1\") " pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:08.905715 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.905511    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lrg\" (UniqueName: \"kubernetes.io/projected/16098cd9-fcba-4c67-bba4-bce8820024a2-kube-api-access-k2lrg\") pod \"ovnkube-node-4dp94\" (UID: \"16098cd9-fcba-4c67-bba4-bce8820024a2\") " pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:08.905867 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.905848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mqw\" (UniqueName: \"kubernetes.io/projected/38668909-95f5-4bbb-84df-f71f4856578a-kube-api-access-k7mqw\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:08.979538 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.979516    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9j4bk"
Sep 11 15:55:08.989520 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:08.989496    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:09.001495 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.001471    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ntxnc"
Sep 11 15:55:09.006717 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.006695    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwf6l"
Sep 11 15:55:09.017035 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.017013    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w5fdn"
Sep 11 15:55:09.025206 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.025187    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:09.033476 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.033455    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-stj8f"
Sep 11 15:55:09.041855 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.041835    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6"
Sep 11 15:55:09.048156 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.048130    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9m89b"
Sep 11 15:55:09.392745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.392672    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:09.392745 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.392720    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:09.392964 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.392753    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:09.392964 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.392825    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:09.392964 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.392849    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:09.392964 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.392866    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:09.392964 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.392877    2564 projected.go:196] Error preparing data for projected volume kube-api-access-585lg for pod openshift-network-diagnostics/network-check-target-k5h9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:09.392964 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.392895    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs podName:38668909-95f5-4bbb-84df-f71f4856578a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:10.392877143 +0000 UTC m=+4.110852258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs") pod "network-metrics-daemon-cfrmq" (UID: "38668909-95f5-4bbb-84df-f71f4856578a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:09.392964 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.392913    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg podName:c0183175-4d09-407f-b523-00a608be58b8 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:10.392903183 +0000 UTC m=+4.110878282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-585lg" (UniqueName: "kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg") pod "network-check-target-k5h9f" (UID: "c0183175-4d09-407f-b523-00a608be58b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:09.392964 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.392920    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:09.393399 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.392984    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret podName:8844b9e8-33aa-4253-8570-bf7b1b1c346e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:10.39296997 +0000 UTC m=+4.110945087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret") pod "global-pull-secret-syncer-74rvc" (UID: "8844b9e8-33aa-4253-8570-bf7b1b1c346e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:09.721469 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.721373    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:50:08 +0000 UTC" deadline="2028-06-04 08:58:41.917915938 +0000 UTC"
Sep 11 15:55:09.721469 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.721414    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15161h3m32.196506698s"
Sep 11 15:55:09.794786 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:09.794764    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682c9904_4b50_46ad_8429_250c436874e6.slice/crio-0b881c6f3dad9c392961b60531f6cc7f9d5bca8f49ef44eee29fc0acbb4771cc WatchSource:0}: Error finding container 0b881c6f3dad9c392961b60531f6cc7f9d5bca8f49ef44eee29fc0acbb4771cc: Status 404 returned error can't find the container with id 0b881c6f3dad9c392961b60531f6cc7f9d5bca8f49ef44eee29fc0acbb4771cc
Sep 11 15:55:09.796035 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:09.796009    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8be5b91_0905_4d4a_83b7_4e80f4b3545e.slice/crio-46ff2cd89aaf7693642acc83add7d2041cb8b312d7bc341df18dbe3b95e14c0d WatchSource:0}: Error finding container 46ff2cd89aaf7693642acc83add7d2041cb8b312d7bc341df18dbe3b95e14c0d: Status 404 returned error can't find the container with id 46ff2cd89aaf7693642acc83add7d2041cb8b312d7bc341df18dbe3b95e14c0d
Sep 11 15:55:09.797058 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:09.797036    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf6c33a_d45b_40b1_a1ba_f42381cd72ad.slice/crio-f585d31807e136aeba0f5b9022d03619c16f88c5463d441beedeed578fea477f WatchSource:0}: Error finding container f585d31807e136aeba0f5b9022d03619c16f88c5463d441beedeed578fea477f: Status 404 returned error can't find the container with id f585d31807e136aeba0f5b9022d03619c16f88c5463d441beedeed578fea477f
Sep 11 15:55:09.798440 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:09.798418    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b4ef1a_6e79_42e2_8f38_6b7355fde851.slice/crio-807a524b3b8fc9f5c23eb356c0daf086e3a2999796445f73059b8a274fdf2b24 WatchSource:0}: Error finding container 807a524b3b8fc9f5c23eb356c0daf086e3a2999796445f73059b8a274fdf2b24: Status 404 returned error can't find the container with id 807a524b3b8fc9f5c23eb356c0daf086e3a2999796445f73059b8a274fdf2b24
Sep 11 15:55:09.801669 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:09.801647    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7165280_6c2a_4719_8420_60faeeac4f82.slice/crio-c6e670ea76070a9d6b306dcfaf43d4e0805358b3b49a813631dbd148caa823a3 WatchSource:0}: Error finding container c6e670ea76070a9d6b306dcfaf43d4e0805358b3b49a813631dbd148caa823a3: Status 404 returned error can't find the container with id c6e670ea76070a9d6b306dcfaf43d4e0805358b3b49a813631dbd148caa823a3
Sep 11 15:55:09.802487 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:09.802378    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16098cd9_fcba_4c67_bba4_bce8820024a2.slice/crio-a7b658d43146a20ab16ce389b7e8f66d4a453ac11af3f5da111bd326f04b2fd3 WatchSource:0}: Error finding container a7b658d43146a20ab16ce389b7e8f66d4a453ac11af3f5da111bd326f04b2fd3: Status 404 returned error can't find the container with id a7b658d43146a20ab16ce389b7e8f66d4a453ac11af3f5da111bd326f04b2fd3
Sep 11 15:55:09.803771 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:09.803751    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c3d20c_209e_4ae8_84d8_5fe3542a4fd1.slice/crio-7ec46bbae914df0223363873af2947610a3683c19af1d0355131b7ae0abb2ee9 WatchSource:0}: Error finding container 7ec46bbae914df0223363873af2947610a3683c19af1d0355131b7ae0abb2ee9: Status 404 returned error can't find the container with id 7ec46bbae914df0223363873af2947610a3683c19af1d0355131b7ae0abb2ee9
Sep 11 15:55:09.804753 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.804673    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:09.804829 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.804803    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:09.804887 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.804850    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:09.804887 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:09.804858    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7e01ee_e3d6_48c6_b567_ad87583a88bc.slice/crio-3bfb35472b86a439e2ef4b3cf80db02f9204ef21812e77b323e8aa48f2deb4a9 WatchSource:0}: Error finding container 3bfb35472b86a439e2ef4b3cf80db02f9204ef21812e77b323e8aa48f2deb4a9: Status 404 returned error can't find the container with id 3bfb35472b86a439e2ef4b3cf80db02f9204ef21812e77b323e8aa48f2deb4a9
Sep 11 15:55:09.804985 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:09.804942    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:09.806816 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:09.806797    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2199b2b_e15f_469b_825a_05423d6fa5ff.slice/crio-4c9dfc37820be439aa889ab57e0a8f487530d641e1b304fdbeb30945c14bb218 WatchSource:0}: Error finding container 4c9dfc37820be439aa889ab57e0a8f487530d641e1b304fdbeb30945c14bb218: Status 404 returned error can't find the container with id 4c9dfc37820be439aa889ab57e0a8f487530d641e1b304fdbeb30945c14bb218
Sep 11 15:55:09.812898 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.812872    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" event={"ID":"16098cd9-fcba-4c67-bba4-bce8820024a2","Type":"ContainerStarted","Data":"a7b658d43146a20ab16ce389b7e8f66d4a453ac11af3f5da111bd326f04b2fd3"}
Sep 11 15:55:09.814866 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.814836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w5fdn" event={"ID":"96b4ef1a-6e79-42e2-8f38-6b7355fde851","Type":"ContainerStarted","Data":"807a524b3b8fc9f5c23eb356c0daf086e3a2999796445f73059b8a274fdf2b24"}
Sep 11 15:55:09.816215 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.816195    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwf6l" event={"ID":"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad","Type":"ContainerStarted","Data":"f585d31807e136aeba0f5b9022d03619c16f88c5463d441beedeed578fea477f"}
Sep 11 15:55:09.817448 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.817418    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ntxnc" event={"ID":"c7165280-6c2a-4719-8420-60faeeac4f82","Type":"ContainerStarted","Data":"c6e670ea76070a9d6b306dcfaf43d4e0805358b3b49a813631dbd148caa823a3"}
Sep 11 15:55:09.819419 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.819379    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9m89b" event={"ID":"b8be5b91-0905-4d4a-83b7-4e80f4b3545e","Type":"ContainerStarted","Data":"46ff2cd89aaf7693642acc83add7d2041cb8b312d7bc341df18dbe3b95e14c0d"}
Sep 11 15:55:09.820661 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.820635    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6" event={"ID":"682c9904-4b50-46ad-8429-250c436874e6","Type":"ContainerStarted","Data":"0b881c6f3dad9c392961b60531f6cc7f9d5bca8f49ef44eee29fc0acbb4771cc"}
Sep 11 15:55:09.821989 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.821963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j4bk" event={"ID":"b2199b2b-e15f-469b-825a-05423d6fa5ff","Type":"ContainerStarted","Data":"4c9dfc37820be439aa889ab57e0a8f487530d641e1b304fdbeb30945c14bb218"}
Sep 11 15:55:09.823786 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.823588    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2nz4w" event={"ID":"8f7e01ee-e3d6-48c6-b567-ad87583a88bc","Type":"ContainerStarted","Data":"3bfb35472b86a439e2ef4b3cf80db02f9204ef21812e77b323e8aa48f2deb4a9"}
Sep 11 15:55:09.824779 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:09.824760    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stj8f" event={"ID":"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1","Type":"ContainerStarted","Data":"7ec46bbae914df0223363873af2947610a3683c19af1d0355131b7ae0abb2ee9"}
Sep 11 15:55:10.403765 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.403678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:10.403765 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.403737    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:10.403970 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.403777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:10.403970 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:10.403905    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:10.403970 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:10.403964    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs podName:38668909-95f5-4bbb-84df-f71f4856578a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:12.403945749 +0000 UTC m=+6.121920856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs") pod "network-metrics-daemon-cfrmq" (UID: "38668909-95f5-4bbb-84df-f71f4856578a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:10.404349 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:10.404199    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:10.404349 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:10.404225    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:10.404349 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:10.404256    2564 projected.go:196] Error preparing data for projected volume kube-api-access-585lg for pod openshift-network-diagnostics/network-check-target-k5h9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:10.404349 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:10.404305    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg podName:c0183175-4d09-407f-b523-00a608be58b8 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:12.4042886 +0000 UTC m=+6.122263714 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-585lg" (UniqueName: "kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg") pod "network-check-target-k5h9f" (UID: "c0183175-4d09-407f-b523-00a608be58b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:10.404630 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:10.404368    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:10.404630 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:10.404412    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret podName:8844b9e8-33aa-4253-8570-bf7b1b1c346e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:12.40440006 +0000 UTC m=+6.122375163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret") pod "global-pull-secret-syncer-74rvc" (UID: "8844b9e8-33aa-4253-8570-bf7b1b1c346e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:10.499552 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.499350    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 15:55:10.721673 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.721561    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 15:50:08 +0000 UTC" deadline="2028-06-17 04:00:07.521323722 +0000 UTC"
Sep 11 15:55:10.721673 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.721620    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15468h4m56.799707516s"
Sep 11 15:55:10.805391 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.805361    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:10.805527 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:10.805486    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:10.862975 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.862938    2564 generic.go:356] "Generic (PLEG): container finished" podID="9d6d8b968f918649697ed3c7789abede" containerID="36b11806f488b0fe69b67cd2be7c0c06d15860cb71316aca6e9217de7bc18948" exitCode=0
Sep 11 15:55:10.863643 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.863535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal" event={"ID":"9d6d8b968f918649697ed3c7789abede","Type":"ContainerDied","Data":"36b11806f488b0fe69b67cd2be7c0c06d15860cb71316aca6e9217de7bc18948"}
Sep 11 15:55:10.873022 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:10.872872    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal" event={"ID":"e8494a3e6a9c170896d58705ab106f0d","Type":"ContainerStarted","Data":"e3cd8e29a2d6752088c1d055f623d9b623892b0aad97e26343af2422cc4d4404"}
Sep 11 15:55:11.805302 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:11.805168    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:11.805981 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:11.805328    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:11.805981 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:11.805348    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:11.805981 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:11.805477    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:11.888571 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:11.888350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal" event={"ID":"9d6d8b968f918649697ed3c7789abede","Type":"ContainerStarted","Data":"831b5b6c2f603812eec8ea6baedbf9e3d3237fcd288435c2ab874c6ea2fb94fb"}
Sep 11 15:55:11.905137 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:11.905086    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-101.ec2.internal" podStartSLOduration=3.905064413 podStartE2EDuration="3.905064413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:10.896661772 +0000 UTC m=+4.614636892" watchObservedRunningTime="2026-09-11 15:55:11.905064413 +0000 UTC m=+5.623039534"
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:12.422795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:12.422847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:12.422881    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:12.422945    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:12.423005    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret podName:8844b9e8-33aa-4253-8570-bf7b1b1c346e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:16.422985711 +0000 UTC m=+10.140960832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret") pod "global-pull-secret-syncer-74rvc" (UID: "8844b9e8-33aa-4253-8570-bf7b1b1c346e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:12.423037    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:12.423072    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs podName:38668909-95f5-4bbb-84df-f71f4856578a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:16.423061634 +0000 UTC m=+10.141036748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs") pod "network-metrics-daemon-cfrmq" (UID: "38668909-95f5-4bbb-84df-f71f4856578a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:12.423075    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:12.423100    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:12.423117 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:12.423114    2564 projected.go:196] Error preparing data for projected volume kube-api-access-585lg for pod openshift-network-diagnostics/network-check-target-k5h9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:12.423779 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:12.423188    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg podName:c0183175-4d09-407f-b523-00a608be58b8 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:16.423147725 +0000 UTC m=+10.141122824 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-585lg" (UniqueName: "kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg") pod "network-check-target-k5h9f" (UID: "c0183175-4d09-407f-b523-00a608be58b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:12.806552 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:12.806001    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:12.806552 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:12.806120    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:13.804712 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:13.804678    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:13.804900 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:13.804821    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:13.804900 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:13.804877    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:13.805010 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:13.804944    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:14.805074 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:14.805039    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:14.805652 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:14.805180    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:15.805191 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:15.805160    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:15.805608 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:15.805307    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:15.805608 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:15.805379    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:15.805608 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:15.805480    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:16.457230 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:16.457192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:16.457419 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:16.457266    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:16.457419 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:16.457306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:16.457419 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:16.457364    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:16.457594 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:16.457423    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:16.457594 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:16.457444    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret podName:8844b9e8-33aa-4253-8570-bf7b1b1c346e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:24.457422988 +0000 UTC m=+18.175398087 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret") pod "global-pull-secret-syncer-74rvc" (UID: "8844b9e8-33aa-4253-8570-bf7b1b1c346e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:16.457594 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:16.457450    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:16.457594 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:16.457472    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:16.457594 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:16.457487    2564 projected.go:196] Error preparing data for projected volume kube-api-access-585lg for pod openshift-network-diagnostics/network-check-target-k5h9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:16.457594 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:16.457471    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs podName:38668909-95f5-4bbb-84df-f71f4856578a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:24.457456344 +0000 UTC m=+18.175431448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs") pod "network-metrics-daemon-cfrmq" (UID: "38668909-95f5-4bbb-84df-f71f4856578a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:16.457594 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:16.457546    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg podName:c0183175-4d09-407f-b523-00a608be58b8 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:24.457532082 +0000 UTC m=+18.175507184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-585lg" (UniqueName: "kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg") pod "network-check-target-k5h9f" (UID: "c0183175-4d09-407f-b523-00a608be58b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:16.806221 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:16.806143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:16.806675 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:16.806267    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:17.805029 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:17.804952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:17.805159 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:17.805076    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:17.805296 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:17.805272    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:17.805460 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:17.805379    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:18.805778 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:18.805454    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:18.805778 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:18.805584    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:19.805163 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:19.805132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:19.805330 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:19.805132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:19.805330 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:19.805234    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:19.805424 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:19.805354    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:20.805453 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:20.805421    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:20.805873 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:20.805542    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:21.805208 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:21.805029    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:21.805387 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:21.805037    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:21.805387 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:21.805300    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:21.805479 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:21.805415    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:22.805333 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:22.805306    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:22.805506 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:22.805409    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:23.805044 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:23.805008    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:23.805307 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:23.805008    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:23.805307 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:23.805138    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:23.805307 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:23.805266    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:24.517820 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:24.517783    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:24.517848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:24.517880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:24.517960    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:24.517978    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:24.517991    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:24.517999    2564 projected.go:196] Error preparing data for projected volume kube-api-access-585lg for pod openshift-network-diagnostics/network-check-target-k5h9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:24.518011    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:24.518028    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs podName:38668909-95f5-4bbb-84df-f71f4856578a nodeName:}" failed. No retries permitted until 2026-09-11 15:55:40.518011098 +0000 UTC m=+34.235986198 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs") pod "network-metrics-daemon-cfrmq" (UID: "38668909-95f5-4bbb-84df-f71f4856578a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:24.518043    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg podName:c0183175-4d09-407f-b523-00a608be58b8 nodeName:}" failed. No retries permitted until 2026-09-11 15:55:40.518036684 +0000 UTC m=+34.236011783 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-585lg" (UniqueName: "kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg") pod "network-check-target-k5h9f" (UID: "c0183175-4d09-407f-b523-00a608be58b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:24.518210 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:24.518056    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret podName:8844b9e8-33aa-4253-8570-bf7b1b1c346e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:40.518049549 +0000 UTC m=+34.236024649 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret") pod "global-pull-secret-syncer-74rvc" (UID: "8844b9e8-33aa-4253-8570-bf7b1b1c346e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:24.805447 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:24.805365    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:24.805613 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:24.805487    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:25.805539 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:25.805503    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:25.805539 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:25.805530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:25.806040 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:25.805618    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:25.806040 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:25.805759    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:26.805611 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:26.805582    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:26.805943 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:26.805660    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:27.805553 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.805387    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:27.805647 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.805423    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:27.806171 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:27.805649    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:27.806171 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:27.805747    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:27.921421 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.921384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9j4bk" event={"ID":"b2199b2b-e15f-469b-825a-05423d6fa5ff","Type":"ContainerStarted","Data":"eef967107e0d7d244042d9db3b3d4a1bf9f30b08323c2328926c7ef3f48836e2"}
Sep 11 15:55:27.923390 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.923360    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2nz4w" event={"ID":"8f7e01ee-e3d6-48c6-b567-ad87583a88bc","Type":"ContainerStarted","Data":"afbe599716ca5922eb180a529fac467d73a3fd104926393bf66ce2784766191d"}
Sep 11 15:55:27.924957 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.924933    2564 generic.go:356] "Generic (PLEG): container finished" podID="86c3d20c-209e-4ae8-84d8-5fe3542a4fd1" containerID="face7eb1557474fcf10bce441960bc8043967088854d0cb735611f87d6bf347c" exitCode=0
Sep 11 15:55:27.925058 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.925021    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stj8f" event={"ID":"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1","Type":"ContainerDied","Data":"face7eb1557474fcf10bce441960bc8043967088854d0cb735611f87d6bf347c"}
Sep 11 15:55:27.929045 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.928820    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" event={"ID":"16098cd9-fcba-4c67-bba4-bce8820024a2","Type":"ContainerStarted","Data":"db515a5c6030ea68a950648d4a519d59308735fa89b08ac0ffb87d874e1d0a71"}
Sep 11 15:55:27.929045 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.928881    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" event={"ID":"16098cd9-fcba-4c67-bba4-bce8820024a2","Type":"ContainerStarted","Data":"e03af74f8a355b5efbf442ddf2cf87ff801e8bcace32a21e775110f2c0a7c973"}
Sep 11 15:55:27.929045 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.928919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" event={"ID":"16098cd9-fcba-4c67-bba4-bce8820024a2","Type":"ContainerStarted","Data":"b11cb0390fcdf563ed3cc14080a6025e3ebbd532022ce4eccf3ee3102a191136"}
Sep 11 15:55:27.929045 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.928962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" event={"ID":"16098cd9-fcba-4c67-bba4-bce8820024a2","Type":"ContainerStarted","Data":"855ad541aec2168a348a75caa3efbffb961c3ddd1e4c4db1d994d1a60fc726d5"}
Sep 11 15:55:27.929045 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.928993    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" event={"ID":"16098cd9-fcba-4c67-bba4-bce8820024a2","Type":"ContainerStarted","Data":"5ea91b81bcdb41f853fbf03945e3bd75f1481f7ca2182e2dbf49a079423a7fc4"}
Sep 11 15:55:27.930479 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.930458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwf6l" event={"ID":"1bf6c33a-d45b-40b1-a1ba-f42381cd72ad","Type":"ContainerStarted","Data":"ec9e093a9d661bfe77d02036a2a05ff698efaf34cee82cd1e532cd7b06930697"}
Sep 11 15:55:27.932301 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.932275    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ntxnc" event={"ID":"c7165280-6c2a-4719-8420-60faeeac4f82","Type":"ContainerStarted","Data":"f0e802698a8f04702f0764d9856173353cc07cbad6941de65ae1737fa9846a1e"}
Sep 11 15:55:27.933925 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.933904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9m89b" event={"ID":"b8be5b91-0905-4d4a-83b7-4e80f4b3545e","Type":"ContainerStarted","Data":"fe942d25ec8d65fbc8d25d49b65aee484a560a944194863b8cf5d248197e80cf"}
Sep 11 15:55:27.935234 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.935213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6" event={"ID":"682c9904-4b50-46ad-8429-250c436874e6","Type":"ContainerStarted","Data":"40a86b73ba5e2a828a6163426c59d5b923614e9144f3e88a304137282e24ce74"}
Sep 11 15:55:27.939410 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.939365    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-101.ec2.internal" podStartSLOduration=19.939348961 podStartE2EDuration="19.939348961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:11.905731425 +0000 UTC m=+5.623706546" watchObservedRunningTime="2026-09-11 15:55:27.939348961 +0000 UTC m=+21.657324081"
Sep 11 15:55:27.939499 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.939442    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9j4bk" podStartSLOduration=3.617101836 podStartE2EDuration="20.939438474s" totalImagesPullingTime="17.322336638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:27.939169102 +0000 UTC m=+21.657144246" watchObservedRunningTime="2026-09-11 15:55:27.939438474 +0000 UTC m=+21.657413593"
Sep 11 15:55:27.956341 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.956306    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9m89b" podStartSLOduration=3.641542402 podStartE2EDuration="20.95629368s" totalImagesPullingTime="17.314751278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:27.9558629 +0000 UTC m=+21.673838041" watchObservedRunningTime="2026-09-11 15:55:27.95629368 +0000 UTC m=+21.674268799"
Sep 11 15:55:27.975588 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.975551    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2nz4w" podStartSLOduration=3.741512376 podStartE2EDuration="20.975539387s" totalImagesPullingTime="17.234027011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:27.974956177 +0000 UTC m=+21.692931298" watchObservedRunningTime="2026-09-11 15:55:27.975539387 +0000 UTC m=+21.693514521"
Sep 11 15:55:27.991996 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:27.991963    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ntxnc" podStartSLOduration=3.682516444 podStartE2EDuration="20.991951363s" totalImagesPullingTime="17.309434919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:27.99185286 +0000 UTC m=+21.709827979" watchObservedRunningTime="2026-09-11 15:55:27.991951363 +0000 UTC m=+21.709926482"
Sep 11 15:55:28.009161 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.009125    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cwf6l" podStartSLOduration=3.739945775 podStartE2EDuration="21.009114075s" totalImagesPullingTime="17.2691683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:28.008974307 +0000 UTC m=+21.726949449" watchObservedRunningTime="2026-09-11 15:55:28.009114075 +0000 UTC m=+21.727089194"
Sep 11 15:55:28.107866 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.107792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:28.151926 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.151898    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:28.178247 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.178221    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:28.462970 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.462939    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 15:55:28.730988 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.730839    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T15:55:28.462960155Z","UUID":"4a1eefb8-8526-4239-baf2-78a6af603865","Handler":null,"Name":"","Endpoint":""}
Sep 11 15:55:28.732459 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.732437    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 15:55:28.732596 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.732473    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 15:55:28.752765 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.752736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:28.805342 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.805320    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:28.805483 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:28.805436    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:28.849654 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.849606    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:28.941033 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.940996    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6" event={"ID":"682c9904-4b50-46ad-8429-250c436874e6","Type":"ContainerStarted","Data":"9a704ca268834c98a1f4edb0e149ef4ecfb30b15e67e68a63872b6bfbc4e5de1"}
Sep 11 15:55:28.944823 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.944790    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" event={"ID":"16098cd9-fcba-4c67-bba4-bce8820024a2","Type":"ContainerStarted","Data":"d8e3ca531de37a0cff5db08e7f99da298dac11a988c93b2a91fdd8ef734b0372"}
Sep 11 15:55:28.946777 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.946747    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w5fdn" event={"ID":"96b4ef1a-6e79-42e2-8f38-6b7355fde851","Type":"ContainerStarted","Data":"f99d3c94fa4fe9ed490188bf9b5b0c7d86f42c4ce2bb1612e7ac1ed3b727fa7c"}
Sep 11 15:55:28.963123 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:28.963081    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w5fdn" podStartSLOduration=4.651781172 podStartE2EDuration="21.963050571s" totalImagesPullingTime="17.311269399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:28.962584712 +0000 UTC m=+22.680559859" watchObservedRunningTime="2026-09-11 15:55:28.963050571 +0000 UTC m=+22.681025691"
Sep 11 15:55:29.089354 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.089268    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:29.150162 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.150102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:29.707413 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.707328    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:29.707985 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.707967    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:29.805047 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.805023    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:29.805213 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.805103    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:29.805213 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:29.805117    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:29.805353 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:29.805308    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:29.880828 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.880790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:29.935542 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.935511    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:29.951660 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.951632    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6" event={"ID":"682c9904-4b50-46ad-8429-250c436874e6","Type":"ContainerStarted","Data":"7b6abad00a296dad7750e179572e966b77608002219d721afd7698e29b20464c"}
Sep 11 15:55:29.952541 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.952517    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2nz4w"
Sep 11 15:55:29.970424 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:29.970360    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c8rs6" podStartSLOduration=3.67057563 podStartE2EDuration="22.970344106s" totalImagesPullingTime="19.299768476s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:29.970078237 +0000 UTC m=+23.688053376" watchObservedRunningTime="2026-09-11 15:55:29.970344106 +0000 UTC m=+23.688319225"
Sep 11 15:55:30.034711 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:30.034685    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:30.281344 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:30.281102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:30.333826 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:30.333793    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:30.804777 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:30.804745    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:30.804970 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:30.804873    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:30.958093 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:30.958060    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" event={"ID":"16098cd9-fcba-4c67-bba4-bce8820024a2","Type":"ContainerStarted","Data":"d3a032a9296991dd9b1bf0112f75d29195ab420235725f7d893127c60529de8c"}
Sep 11 15:55:31.088297 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:31.088200    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:31.143845 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:31.143808    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:31.493473 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:31.493393    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:31.563749 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:31.563715    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:31.805548 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:31.805478    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:31.805548 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:31.805515    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:31.805744 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:31.805605    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:31.805744 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:31.805703    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:32.097774 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.097745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:32.193319 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.193293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:32.805434 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.805164    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:32.805556 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:32.805485    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:32.964264 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.964224    2564 generic.go:356] "Generic (PLEG): container finished" podID="86c3d20c-209e-4ae8-84d8-5fe3542a4fd1" containerID="a643bbb360fa5c7044a94f3e7a2e05d2b34369c9cdd565857e0a77f42d4b5cb5" exitCode=0
Sep 11 15:55:32.964423 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.964312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stj8f" event={"ID":"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1","Type":"ContainerDied","Data":"a643bbb360fa5c7044a94f3e7a2e05d2b34369c9cdd565857e0a77f42d4b5cb5"}
Sep 11 15:55:32.968224 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.968203    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" event={"ID":"16098cd9-fcba-4c67-bba4-bce8820024a2","Type":"ContainerStarted","Data":"8b1319c1188e52ecea3bbd48812a800dda6ff0bdffbedc60fb0cfd49ad15fa44"}
Sep 11 15:55:32.968482 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.968466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:32.968588 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.968510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:32.968588 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.968560    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:32.983221 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.983203    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:32.983401 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:32.983384    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:55:33.026051 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:33.026006    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94" podStartSLOduration=8.435382764 podStartE2EDuration="26.02599078s" totalImagesPullingTime="17.590608016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:33.025650919 +0000 UTC m=+26.743626076" watchObservedRunningTime="2026-09-11 15:55:33.02599078 +0000 UTC m=+26.743965900"
Sep 11 15:55:33.805393 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:33.805364    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:33.805852 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:33.805363    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:33.805852 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:33.805489    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:33.805852 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:33.805531    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:34.168200 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:34.168172    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-74rvc"]
Sep 11 15:55:34.168388 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:34.168327    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:34.168469 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:34.168426    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:34.168914 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:34.168895    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k5h9f"]
Sep 11 15:55:34.168998 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:34.168972    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:34.169053 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:34.169037    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:34.170687 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:34.170665    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cfrmq"]
Sep 11 15:55:34.170792 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:34.170771    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:34.170906 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:34.170884    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:34.974506 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:34.974469    2564 generic.go:356] "Generic (PLEG): container finished" podID="86c3d20c-209e-4ae8-84d8-5fe3542a4fd1" containerID="4e6c01b9ec2d3b2cac75c410ad61230cfdfd289557bf2f1b7a960dc3f4e2007f" exitCode=0
Sep 11 15:55:34.975510 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:34.974549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stj8f" event={"ID":"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1","Type":"ContainerDied","Data":"4e6c01b9ec2d3b2cac75c410ad61230cfdfd289557bf2f1b7a960dc3f4e2007f"}
Sep 11 15:55:35.805167 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:35.805134    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:35.805334 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:35.805134    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:35.805334 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:35.805250    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:35.805334 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:35.805135    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:35.805334 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:35.805307    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:35.805485 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:35.805374    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:36.688916 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:36.688695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:36.756192 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:36.756158    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:36.980731 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:36.980701    2564 generic.go:356] "Generic (PLEG): container finished" podID="86c3d20c-209e-4ae8-84d8-5fe3542a4fd1" containerID="cef8d6f646a2e138ce7bfcb369611c4622235ccd66905db8039153fec6fafc17" exitCode=0
Sep 11 15:55:36.980841 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:36.980752    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stj8f" event={"ID":"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1","Type":"ContainerDied","Data":"cef8d6f646a2e138ce7bfcb369611c4622235ccd66905db8039153fec6fafc17"}
Sep 11 15:55:37.522069 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:37.522035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:37.624543 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:37.624507    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:37.736090 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:37.736054    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:37.805494 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:37.805423    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:37.805494 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:37.805436    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:37.805494 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:37.805440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:37.805712 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:37.805535    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:37.805712 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:37.805608    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:37.805712 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:37.805687    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:37.837506 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:37.837476    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:37.906307 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:37.906276    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:38.166430 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:38.166340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:38.267228 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:38.267195    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:38.488128 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:38.488029    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:38.594059 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:38.594028    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:39.031631 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:39.031604    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9m89b_b8be5b91-0905-4d4a-83b7-4e80f4b3545e/dns-node-resolver/0.log"
Sep 11 15:55:39.766361 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:39.766319    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:39.804544 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:39.804517    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:39.804696 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:39.804554    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:39.804696 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:39.804554    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:39.804696 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:39.804636    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-74rvc" podUID="8844b9e8-33aa-4253-8570-bf7b1b1c346e"
Sep 11 15:55:39.804857 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:39.804745    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cfrmq" podUID="38668909-95f5-4bbb-84df-f71f4856578a"
Sep 11 15:55:39.804857 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:39.804799    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5h9f" podUID="c0183175-4d09-407f-b523-00a608be58b8"
Sep 11 15:55:39.865142 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:39.865118    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:40.025714 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.025630    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-101.ec2.internal" event="NodeReady"
Sep 11 15:55:40.025846 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.025754    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 15:55:40.031295 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.031275    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwf6l_1bf6c33a-d45b-40b1-a1ba-f42381cd72ad/node-ca/0.log"
Sep 11 15:55:40.096259 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.096214    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-t48lx"]
Sep 11 15:55:40.099117 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.099093    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ncw2l"]
Sep 11 15:55:40.099227 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.099211    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.100961 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.100937    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.101899 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.101876    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 15:55:40.102003 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.101876    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jvv4c\""
Sep 11 15:55:40.102077 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.102001    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 15:55:40.103537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.103517    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 15:55:40.103537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.103532    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 15:55:40.103885 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.103867    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 15:55:40.103976 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.103898    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 15:55:40.103976 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.103898    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4ddf\""
Sep 11 15:55:40.108281 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.108251    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:40.115175 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.115153    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ncw2l"]
Sep 11 15:55:40.115777 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.115758    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t48lx"]
Sep 11 15:55:40.120918 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.120895    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-26dlc"]
Sep 11 15:55:40.122603 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.122588    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-26dlc"
Sep 11 15:55:40.125341 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.125321    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dxw2k\""
Sep 11 15:55:40.126443 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.126421    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 15:55:40.126688 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.126672    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 15:55:40.126770 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.126723    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 15:55:40.133284 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.133251    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-26dlc"]
Sep 11 15:55:40.178656 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.178580    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:40.237076 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237047    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-962gk\" (UniqueName: \"kubernetes.io/projected/96390e07-591b-4c7a-bbe7-fe4b350f254e-kube-api-access-962gk\") pod \"ingress-canary-26dlc\" (UID: \"96390e07-591b-4c7a-bbe7-fe4b350f254e\") " pod="openshift-ingress-canary/ingress-canary-26dlc"
Sep 11 15:55:40.237191 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/495d4158-a486-437d-9a76-1de6cca00d7e-config-volume\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.237191 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcsw4\" (UniqueName: \"kubernetes.io/projected/495d4158-a486-437d-9a76-1de6cca00d7e-kube-api-access-mcsw4\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.237191 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d4158-a486-437d-9a76-1de6cca00d7e-tmp-dir\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.237381 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237201    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb9zt\" (UniqueName: \"kubernetes.io/projected/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-api-access-kb9zt\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.237381 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.237381 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237338    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dc268f-8a0d-4d2c-add9-532213e53a5b-data-volume\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.237547 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237396    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.237547 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96390e07-591b-4c7a-bbe7-fe4b350f254e-cert\") pod \"ingress-canary-26dlc\" (UID: \"96390e07-591b-4c7a-bbe7-fe4b350f254e\") " pod="openshift-ingress-canary/ingress-canary-26dlc"
Sep 11 15:55:40.237547 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5dc268f-8a0d-4d2c-add9-532213e53a5b-crio-socket\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.237547 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.237498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.251730 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.251706    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:40.337718 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.337695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.337816 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.337729    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-962gk\" (UniqueName: \"kubernetes.io/projected/96390e07-591b-4c7a-bbe7-fe4b350f254e-kube-api-access-962gk\") pod \"ingress-canary-26dlc\" (UID: \"96390e07-591b-4c7a-bbe7-fe4b350f254e\") " pod="openshift-ingress-canary/ingress-canary-26dlc"
Sep 11 15:55:40.337816 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.337752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/495d4158-a486-437d-9a76-1de6cca00d7e-config-volume\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.337899 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.337842    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:40.337899 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.337873    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcsw4\" (UniqueName: \"kubernetes.io/projected/495d4158-a486-437d-9a76-1de6cca00d7e-kube-api-access-mcsw4\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.337955 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.337917    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls podName:495d4158-a486-437d-9a76-1de6cca00d7e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:40.837898197 +0000 UTC m=+34.555873305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls") pod "dns-default-t48lx" (UID: "495d4158-a486-437d-9a76-1de6cca00d7e") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:40.337994 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.337969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d4158-a486-437d-9a76-1de6cca00d7e-tmp-dir\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.338026 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338005    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb9zt\" (UniqueName: \"kubernetes.io/projected/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-api-access-kb9zt\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.338058 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.338194 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dc268f-8a0d-4d2c-add9-532213e53a5b-data-volume\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.338263 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.338231    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 15:55:40.338263 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.338362 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.338311    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm podName:d5dc268f-8a0d-4d2c-add9-532213e53a5b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:40.838292936 +0000 UTC m=+34.556268053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncw2l" (UID: "d5dc268f-8a0d-4d2c-add9-532213e53a5b") : configmap "kube-rbac-proxy" not found
Sep 11 15:55:40.338362 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338337    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d4158-a486-437d-9a76-1de6cca00d7e-tmp-dir\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.338362 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96390e07-591b-4c7a-bbe7-fe4b350f254e-cert\") pod \"ingress-canary-26dlc\" (UID: \"96390e07-591b-4c7a-bbe7-fe4b350f254e\") " pod="openshift-ingress-canary/ingress-canary-26dlc"
Sep 11 15:55:40.338362 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.338358    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:40.338552 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338361    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/495d4158-a486-437d-9a76-1de6cca00d7e-config-volume\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.338552 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5dc268f-8a0d-4d2c-add9-532213e53a5b-crio-socket\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.338552 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.338403    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls podName:d5dc268f-8a0d-4d2c-add9-532213e53a5b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:40.838387571 +0000 UTC m=+34.556362683 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncw2l" (UID: "d5dc268f-8a0d-4d2c-add9-532213e53a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:40.338678 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d5dc268f-8a0d-4d2c-add9-532213e53a5b-data-volume\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.338678 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.338565    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d5dc268f-8a0d-4d2c-add9-532213e53a5b-crio-socket\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.342940 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.342917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96390e07-591b-4c7a-bbe7-fe4b350f254e-cert\") pod \"ingress-canary-26dlc\" (UID: \"96390e07-591b-4c7a-bbe7-fe4b350f254e\") " pod="openshift-ingress-canary/ingress-canary-26dlc"
Sep 11 15:55:40.348656 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.348633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcsw4\" (UniqueName: \"kubernetes.io/projected/495d4158-a486-437d-9a76-1de6cca00d7e-kube-api-access-mcsw4\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.348765 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.348677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-962gk\" (UniqueName: \"kubernetes.io/projected/96390e07-591b-4c7a-bbe7-fe4b350f254e-kube-api-access-962gk\") pod \"ingress-canary-26dlc\" (UID: \"96390e07-591b-4c7a-bbe7-fe4b350f254e\") " pod="openshift-ingress-canary/ingress-canary-26dlc"
Sep 11 15:55:40.348765 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.348680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb9zt\" (UniqueName: \"kubernetes.io/projected/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-api-access-kb9zt\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.352656 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.352629    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:40.431886 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.431865    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-26dlc"
Sep 11 15:55:40.540381 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.540346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:40.540381 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.540391    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:40.540599 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.540434    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:40.540599 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.540518    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:40.540599 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.540554    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 15:55:40.540599 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.540572    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 15:55:40.540599 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.540583    2564 projected.go:196] Error preparing data for projected volume kube-api-access-585lg for pod openshift-network-diagnostics/network-check-target-k5h9f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:40.540599 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.540589    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret podName:8844b9e8-33aa-4253-8570-bf7b1b1c346e nodeName:}" failed. No retries permitted until 2026-09-11 15:56:12.540569365 +0000 UTC m=+66.258544467 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret") pod "global-pull-secret-syncer-74rvc" (UID: "8844b9e8-33aa-4253-8570-bf7b1b1c346e") : object "kube-system"/"original-pull-secret" not registered
Sep 11 15:55:40.540820 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.540612    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:40.540820 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.540631    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg podName:c0183175-4d09-407f-b523-00a608be58b8 nodeName:}" failed. No retries permitted until 2026-09-11 15:56:12.540620805 +0000 UTC m=+66.258595903 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-585lg" (UniqueName: "kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg") pod "network-check-target-k5h9f" (UID: "c0183175-4d09-407f-b523-00a608be58b8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 15:55:40.540820 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.540648    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs podName:38668909-95f5-4bbb-84df-f71f4856578a nodeName:}" failed. No retries permitted until 2026-09-11 15:56:12.540637053 +0000 UTC m=+66.258612168 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs") pod "network-metrics-daemon-cfrmq" (UID: "38668909-95f5-4bbb-84df-f71f4856578a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 15:55:40.842567 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.842437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.842686 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.842598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:40.842686 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.842627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:40.842686 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.842549    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:40.842686 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.842672    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:40.842803 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.842691    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls podName:d5dc268f-8a0d-4d2c-add9-532213e53a5b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:41.842679447 +0000 UTC m=+35.560654558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncw2l" (UID: "d5dc268f-8a0d-4d2c-add9-532213e53a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:40.842803 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.842693    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 15:55:40.842803 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.842714    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls podName:495d4158-a486-437d-9a76-1de6cca00d7e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:41.842702557 +0000 UTC m=+35.560677655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls") pod "dns-default-t48lx" (UID: "495d4158-a486-437d-9a76-1de6cca00d7e") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:40.842803 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:40.842735    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm podName:d5dc268f-8a0d-4d2c-add9-532213e53a5b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:41.842724322 +0000 UTC m=+35.560699420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncw2l" (UID: "d5dc268f-8a0d-4d2c-add9-532213e53a5b") : configmap "kube-rbac-proxy" not found
Sep 11 15:55:40.992328 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.992267    2564 generic.go:356] "Generic (PLEG): container finished" podID="86c3d20c-209e-4ae8-84d8-5fe3542a4fd1" containerID="99adb8bdd9538f24c1f19e2101304ea98cc0b9437db77971c2ee4a44a45e1570" exitCode=0
Sep 11 15:55:40.992431 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:40.992331    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stj8f" event={"ID":"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1","Type":"ContainerDied","Data":"99adb8bdd9538f24c1f19e2101304ea98cc0b9437db77971c2ee4a44a45e1570"}
Sep 11 15:55:41.022428 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.022400    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-26dlc"]
Sep 11 15:55:41.026219 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:41.026192    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96390e07_591b_4c7a_bbe7_fe4b350f254e.slice/crio-fd9f9456b41e20d3c827b2f0ec5137ef33a6281f5480989f6e68a31cf350bffa WatchSource:0}: Error finding container fd9f9456b41e20d3c827b2f0ec5137ef33a6281f5480989f6e68a31cf350bffa: Status 404 returned error can't find the container with id fd9f9456b41e20d3c827b2f0ec5137ef33a6281f5480989f6e68a31cf350bffa
Sep 11 15:55:41.333910 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.333881    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:41.805339 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.805236    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:55:41.805339 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.805287    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:55:41.805549 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.805257    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:55:41.808364 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.808344    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 15:55:41.809669 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.809645    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lsspg\""
Sep 11 15:55:41.809791 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.809676    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 15:55:41.809791 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.809690    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 15:55:41.809791 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.809754    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zsbjs\""
Sep 11 15:55:41.809943 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.809819    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 15:55:41.851054 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.851033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:41.851137 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.851077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:41.851197 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:41.851155    2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 15:55:41.851197 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:41.851175    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:41.851324 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:41.851204    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm podName:d5dc268f-8a0d-4d2c-add9-532213e53a5b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:43.851187131 +0000 UTC m=+37.569162236 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ncw2l" (UID: "d5dc268f-8a0d-4d2c-add9-532213e53a5b") : configmap "kube-rbac-proxy" not found
Sep 11 15:55:41.851324 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:41.851229    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls podName:495d4158-a486-437d-9a76-1de6cca00d7e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:43.851212077 +0000 UTC m=+37.569187194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls") pod "dns-default-t48lx" (UID: "495d4158-a486-437d-9a76-1de6cca00d7e") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:41.851324 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.851310    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:41.851451 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:41.851402    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:41.851487 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:41.851459    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls podName:d5dc268f-8a0d-4d2c-add9-532213e53a5b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:43.851448406 +0000 UTC m=+37.569423504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncw2l" (UID: "d5dc268f-8a0d-4d2c-add9-532213e53a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:41.996362 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.996329    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-26dlc" event={"ID":"96390e07-591b-4c7a-bbe7-fe4b350f254e","Type":"ContainerStarted","Data":"fd9f9456b41e20d3c827b2f0ec5137ef33a6281f5480989f6e68a31cf350bffa"}
Sep 11 15:55:41.999891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.999867    2564 generic.go:356] "Generic (PLEG): container finished" podID="86c3d20c-209e-4ae8-84d8-5fe3542a4fd1" containerID="570caf0e89dc7a7d8250f99422cba60590aeca982252cb34bb35666db2066659" exitCode=0
Sep 11 15:55:41.999979 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:41.999898    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stj8f" event={"ID":"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1","Type":"ContainerDied","Data":"570caf0e89dc7a7d8250f99422cba60590aeca982252cb34bb35666db2066659"}
Sep 11 15:55:43.007094 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:43.007059    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-stj8f" event={"ID":"86c3d20c-209e-4ae8-84d8-5fe3542a4fd1","Type":"ContainerStarted","Data":"405f0054e14237c50d7dab63edef2955854c9f4c886674579cdbdc4d31c387b9"}
Sep 11 15:55:43.033939 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:43.033885    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-stj8f" podStartSLOduration=8.736092486 podStartE2EDuration="36.033865782s" totalImagesPullingTime="27.297773296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:43.031961651 +0000 UTC m=+36.749936789" watchObservedRunningTime="2026-09-11 15:55:43.033865782 +0000 UTC m=+36.751840902"
Sep 11 15:55:43.046589 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:43.046554    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:43.112716 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:43.112692    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:43.868915 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:43.868885    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:43.869060 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:43.868924    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:43.869060 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:43.868968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:43.869060 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:43.869035    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:43.869060 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:43.869048    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:43.869234 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:43.869099    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls podName:495d4158-a486-437d-9a76-1de6cca00d7e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:47.869083178 +0000 UTC m=+41.587058281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls") pod "dns-default-t48lx" (UID: "495d4158-a486-437d-9a76-1de6cca00d7e") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:43.869234 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:43.869113    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls podName:d5dc268f-8a0d-4d2c-add9-532213e53a5b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:47.869107768 +0000 UTC m=+41.587082866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncw2l" (UID: "d5dc268f-8a0d-4d2c-add9-532213e53a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:43.869484 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:43.869464    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d5dc268f-8a0d-4d2c-add9-532213e53a5b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:44.011290 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:44.011213    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-26dlc" event={"ID":"96390e07-591b-4c7a-bbe7-fe4b350f254e","Type":"ContainerStarted","Data":"81ce0cbd65e3f32e046a8c0d85c6eab12e5e6799b2ed95406dd5e42be362e6e1"}
Sep 11 15:55:44.038959 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:44.038915    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-26dlc" podStartSLOduration=1.813570106 podStartE2EDuration="4.038901901s" totalImagesPullingTime="2.225331795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:44.038632572 +0000 UTC m=+37.756607711" watchObservedRunningTime="2026-09-11 15:55:44.038901901 +0000 UTC m=+37.756877014"
Sep 11 15:55:44.935779 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:44.935606    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:45.171150 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:45.171121    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:45.274917 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:45.274827    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:45.407824 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:45.407797    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:45.503962 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:45.503932    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:45.570464 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:45.570436    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:45.676174 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:45.676145    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:45.781340 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:45.781301    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:45.876541 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:45.876468    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:47.080525 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:47.080484    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:47.277749 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:47.277711    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:47.487943 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:47.487864    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:47.647520 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:47.647486    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:47.840697 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:47.840664    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:47.898597 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:47.898573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:47.898674 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:47.898617    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:47.898733 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:47.898717    2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:47.898786 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:47.898776    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls podName:d5dc268f-8a0d-4d2c-add9-532213e53a5b nodeName:}" failed. No retries permitted until 2026-09-11 15:55:55.898761823 +0000 UTC m=+49.616736922 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ncw2l" (UID: "d5dc268f-8a0d-4d2c-add9-532213e53a5b") : secret "insights-runtime-extractor-tls" not found
Sep 11 15:55:47.898833 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:47.898722    2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 15:55:47.898862 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:55:47.898856    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls podName:495d4158-a486-437d-9a76-1de6cca00d7e nodeName:}" failed. No retries permitted until 2026-09-11 15:55:55.898843119 +0000 UTC m=+49.616818217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls") pod "dns-default-t48lx" (UID: "495d4158-a486-437d-9a76-1de6cca00d7e") : secret "dns-default-metrics-tls" not found
Sep 11 15:55:48.645315 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:48.645280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:48.745621 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:48.745588    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:48.847629 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:48.847599    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:49.041846 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:49.041784    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:49.247958 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:49.247921    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:49.351918 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:49.351891    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:49.866920 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:49.866883    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:50.239036 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:50.238966    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:50.639052 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:50.639026    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:50.739674 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:50.739651    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:50.800509 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:50.800482    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:50.869071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:50.869045    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:51.355034 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:51.355003    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:51.455925 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:51.455893    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:52.038919 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:52.038885    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:52.244845 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:52.244815    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:52.437496 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:52.437462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:52.538445 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:52.538411    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:52.642219 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:52.642190    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:53.674301 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:53.674265    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:54.043270 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:54.043181    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:54.244811 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:54.244785    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:54.445007 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:54.444976    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:54.646510 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:54.646477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:54.883980 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:54.883947    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:54.950467 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:54.950439    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:55.241138 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:55.241048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:55.652641 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:55.652612    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:55.846419 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:55.846385    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:55.952817 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:55.952744    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:55.952817 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:55.952801    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:55.956025 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:55.955999    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/495d4158-a486-437d-9a76-1de6cca00d7e-metrics-tls\") pod \"dns-default-t48lx\" (UID: \"495d4158-a486-437d-9a76-1de6cca00d7e\") " pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:55.956025 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:55.955999    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d5dc268f-8a0d-4d2c-add9-532213e53a5b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ncw2l\" (UID: \"d5dc268f-8a0d-4d2c-add9-532213e53a5b\") " pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:56.011021 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:56.011000    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:56.018178 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:56.018137    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ncw2l"
Sep 11 15:55:56.320722 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:56.320694    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-t48lx"]
Sep 11 15:55:56.323629 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:56.323601    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495d4158_a486_437d_9a76_1de6cca00d7e.slice/crio-87f1c6be0c57d603994a53ced1228ca46d2d461a18952f566bb1a812dbde669a WatchSource:0}: Error finding container 87f1c6be0c57d603994a53ced1228ca46d2d461a18952f566bb1a812dbde669a: Status 404 returned error can't find the container with id 87f1c6be0c57d603994a53ced1228ca46d2d461a18952f566bb1a812dbde669a
Sep 11 15:55:56.347234 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:56.347212    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ncw2l"]
Sep 11 15:55:56.350105 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:55:56.350083    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5dc268f_8a0d_4d2c_add9_532213e53a5b.slice/crio-86da3781d426b726e7a97f4fa6d46450794cd258bfcd8f6a2ded98ef76441b5e WatchSource:0}: Error finding container 86da3781d426b726e7a97f4fa6d46450794cd258bfcd8f6a2ded98ef76441b5e: Status 404 returned error can't find the container with id 86da3781d426b726e7a97f4fa6d46450794cd258bfcd8f6a2ded98ef76441b5e
Sep 11 15:55:57.040969 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:57.040933    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncw2l" event={"ID":"d5dc268f-8a0d-4d2c-add9-532213e53a5b","Type":"ContainerStarted","Data":"6ae6ef5e2efdd937911891ae17f81b5430c3937431fa45632377a1c25e4f4cda"}
Sep 11 15:55:57.041429 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:57.041000    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncw2l" event={"ID":"d5dc268f-8a0d-4d2c-add9-532213e53a5b","Type":"ContainerStarted","Data":"86da3781d426b726e7a97f4fa6d46450794cd258bfcd8f6a2ded98ef76441b5e"}
Sep 11 15:55:57.042171 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:57.042135    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t48lx" event={"ID":"495d4158-a486-437d-9a76-1de6cca00d7e","Type":"ContainerStarted","Data":"87f1c6be0c57d603994a53ced1228ca46d2d461a18952f566bb1a812dbde669a"}
Sep 11 15:55:59.049563 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.049521    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncw2l" event={"ID":"d5dc268f-8a0d-4d2c-add9-532213e53a5b","Type":"ContainerStarted","Data":"e2cbf1de282bbb0629fc5e8538d0d7a230ad681645521d761444b1674b0fbbf9"}
Sep 11 15:55:59.051388 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.051359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t48lx" event={"ID":"495d4158-a486-437d-9a76-1de6cca00d7e","Type":"ContainerStarted","Data":"dc59c001ebeb701c656d938cb8559cbb530e0063bd51f93eb2d6a4249cfd6157"}
Sep 11 15:55:59.051498 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.051414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-t48lx" event={"ID":"495d4158-a486-437d-9a76-1de6cca00d7e","Type":"ContainerStarted","Data":"c1f361e7ec84e5e14a36135e22e4e86e8d72b8369f17122fde4890612316cbcd"}
Sep 11 15:55:59.051498 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.051470    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-t48lx"
Sep 11 15:55:59.069978 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.069929    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-t48lx" podStartSLOduration=17.099988839 podStartE2EDuration="19.069911322s" totalImagesPullingTime="1.969922483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:55:59.068971659 +0000 UTC m=+52.786946779" watchObservedRunningTime="2026-09-11 15:55:59.069911322 +0000 UTC m=+52.787886443"
Sep 11 15:55:59.573785 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.573751    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"]
Sep 11 15:55:59.575547 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.575530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"
Sep 11 15:55:59.578137 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.578116    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-fglf9\""
Sep 11 15:55:59.578416 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.578395    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 15:55:59.579671 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.579654    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 15:55:59.579671 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.579663    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 15:55:59.579880 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.579835    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 15:55:59.585635 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.585612    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"]
Sep 11 15:55:59.587575 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.587559    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"]
Sep 11 15:55:59.587696 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.587682    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.589183 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.589167    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.592104 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.592074    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"]
Sep 11 15:55:59.592528 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.592510    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 15:55:59.592630 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.592510    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 15:55:59.592717 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.592697    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 15:55:59.592824 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.592810    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 15:55:59.596358 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.596300    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 15:55:59.606690 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.606669    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"]
Sep 11 15:55:59.610743 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.610699    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"]
Sep 11 15:55:59.679708 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.679685    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be1b7ea3-3ed2-4a1d-909c-189b398f65c9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-864bd58df6-wj5mg\" (UID: \"be1b7ea3-3ed2-4a1d-909c-189b398f65c9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"
Sep 11 15:55:59.679810 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.679729    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8gp\" (UniqueName: \"kubernetes.io/projected/be1b7ea3-3ed2-4a1d-909c-189b398f65c9-kube-api-access-lm8gp\") pod \"managed-serviceaccount-addon-agent-864bd58df6-wj5mg\" (UID: \"be1b7ea3-3ed2-4a1d-909c-189b398f65c9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"
Sep 11 15:55:59.719418 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.719395    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:59.780254 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780221    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cfa70cb1-50f4-491d-a6b2-dd5b1372987f-klusterlet-config\") pod \"klusterlet-addon-workmgr-79cdcb5f84-4nz5g\" (UID: \"cfa70cb1-50f4-491d-a6b2-dd5b1372987f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.780341 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780270    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.780341 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.780341 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780319    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be1b7ea3-3ed2-4a1d-909c-189b398f65c9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-864bd58df6-wj5mg\" (UID: \"be1b7ea3-3ed2-4a1d-909c-189b398f65c9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"
Sep 11 15:55:59.780472 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-hub\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.780472 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxqfr\" (UniqueName: \"kubernetes.io/projected/9a850363-389b-4e2f-8aa7-aaeaaca36488-kube-api-access-kxqfr\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.780472 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780403    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjqkp\" (UniqueName: \"kubernetes.io/projected/cfa70cb1-50f4-491d-a6b2-dd5b1372987f-kube-api-access-cjqkp\") pod \"klusterlet-addon-workmgr-79cdcb5f84-4nz5g\" (UID: \"cfa70cb1-50f4-491d-a6b2-dd5b1372987f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.780472 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780453    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8gp\" (UniqueName: \"kubernetes.io/projected/be1b7ea3-3ed2-4a1d-909c-189b398f65c9-kube-api-access-lm8gp\") pod \"managed-serviceaccount-addon-agent-864bd58df6-wj5mg\" (UID: \"be1b7ea3-3ed2-4a1d-909c-189b398f65c9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"
Sep 11 15:55:59.780623 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a850363-389b-4e2f-8aa7-aaeaaca36488-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.780733 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780495    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfa70cb1-50f4-491d-a6b2-dd5b1372987f-tmp\") pod \"klusterlet-addon-workmgr-79cdcb5f84-4nz5g\" (UID: \"cfa70cb1-50f4-491d-a6b2-dd5b1372987f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.780878 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.780849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-ca\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.784134 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.783647    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be1b7ea3-3ed2-4a1d-909c-189b398f65c9-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-864bd58df6-wj5mg\" (UID: \"be1b7ea3-3ed2-4a1d-909c-189b398f65c9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"
Sep 11 15:55:59.793120 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.793099    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8gp\" (UniqueName: \"kubernetes.io/projected/be1b7ea3-3ed2-4a1d-909c-189b398f65c9-kube-api-access-lm8gp\") pod \"managed-serviceaccount-addon-agent-864bd58df6-wj5mg\" (UID: \"be1b7ea3-3ed2-4a1d-909c-189b398f65c9\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"
Sep 11 15:55:59.805730 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.805705    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:59.881966 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.881922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-hub\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.881966 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.881945    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxqfr\" (UniqueName: \"kubernetes.io/projected/9a850363-389b-4e2f-8aa7-aaeaaca36488-kube-api-access-kxqfr\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.881966 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.881962    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjqkp\" (UniqueName: \"kubernetes.io/projected/cfa70cb1-50f4-491d-a6b2-dd5b1372987f-kube-api-access-cjqkp\") pod \"klusterlet-addon-workmgr-79cdcb5f84-4nz5g\" (UID: \"cfa70cb1-50f4-491d-a6b2-dd5b1372987f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.882163 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.881980    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a850363-389b-4e2f-8aa7-aaeaaca36488-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.882163 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.882104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfa70cb1-50f4-491d-a6b2-dd5b1372987f-tmp\") pod \"klusterlet-addon-workmgr-79cdcb5f84-4nz5g\" (UID: \"cfa70cb1-50f4-491d-a6b2-dd5b1372987f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.882304 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.882214    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-ca\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.882304 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.882264    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cfa70cb1-50f4-491d-a6b2-dd5b1372987f-klusterlet-config\") pod \"klusterlet-addon-workmgr-79cdcb5f84-4nz5g\" (UID: \"cfa70cb1-50f4-491d-a6b2-dd5b1372987f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.882421 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.882402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.882474 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.882439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.882552 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.882522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfa70cb1-50f4-491d-a6b2-dd5b1372987f-tmp\") pod \"klusterlet-addon-workmgr-79cdcb5f84-4nz5g\" (UID: \"cfa70cb1-50f4-491d-a6b2-dd5b1372987f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.882856 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.882835    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a850363-389b-4e2f-8aa7-aaeaaca36488-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.884545 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.884512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-hub\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.884994 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.884969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-ca\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.885089 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.885075    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cfa70cb1-50f4-491d-a6b2-dd5b1372987f-klusterlet-config\") pod \"klusterlet-addon-workmgr-79cdcb5f84-4nz5g\" (UID: \"cfa70cb1-50f4-491d-a6b2-dd5b1372987f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.885127 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.885084    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.885177 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.885162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a850363-389b-4e2f-8aa7-aaeaaca36488-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.891849 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.891825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxqfr\" (UniqueName: \"kubernetes.io/projected/9a850363-389b-4e2f-8aa7-aaeaaca36488-kube-api-access-kxqfr\") pod \"cluster-proxy-proxy-agent-7d44595964-zb47v\" (UID: \"9a850363-389b-4e2f-8aa7-aaeaaca36488\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.892174 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.892151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjqkp\" (UniqueName: \"kubernetes.io/projected/cfa70cb1-50f4-491d-a6b2-dd5b1372987f-kube-api-access-cjqkp\") pod \"klusterlet-addon-workmgr-79cdcb5f84-4nz5g\" (UID: \"cfa70cb1-50f4-491d-a6b2-dd5b1372987f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:55:59.897736 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.897722    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"
Sep 11 15:55:59.904698 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.904671    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"
Sep 11 15:55:59.905724 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.905704    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:55:59.920252 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:55:59.920214    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:56:00.028611 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:00.028579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:00.056420 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:00.056395    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ncw2l" event={"ID":"d5dc268f-8a0d-4d2c-add9-532213e53a5b","Type":"ContainerStarted","Data":"bcf6fb6ed486a918493fe7ced26f12dd46109b3fa536857206488b4df490be58"}
Sep 11 15:56:00.131324 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:00.131305    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:00.176169 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:00.176104    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ncw2l" podStartSLOduration=17.391338935 podStartE2EDuration="20.176090134s" totalImagesPullingTime="2.784751199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:00.08292625 +0000 UTC m=+53.800901386" watchObservedRunningTime="2026-09-11 15:56:00.176090134 +0000 UTC m=+53.894065254"
Sep 11 15:56:00.176721 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:00.176679    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg"]
Sep 11 15:56:00.179819 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:56:00.179796    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe1b7ea3_3ed2_4a1d_909c_189b398f65c9.slice/crio-2922a3d220c7b7616803750b48af97d877136f9a2862a6aace6435b27cf41af5 WatchSource:0}: Error finding container 2922a3d220c7b7616803750b48af97d877136f9a2862a6aace6435b27cf41af5: Status 404 returned error can't find the container with id 2922a3d220c7b7616803750b48af97d877136f9a2862a6aace6435b27cf41af5
Sep 11 15:56:00.180369 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:00.180347    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v"]
Sep 11 15:56:00.182316 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:56:00.182294    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a850363_389b_4e2f_8aa7_aaeaaca36488.slice/crio-63941436dd22c376e05d022560844255ffc2c95835cc17b3638cb536dff0e6b1 WatchSource:0}: Error finding container 63941436dd22c376e05d022560844255ffc2c95835cc17b3638cb536dff0e6b1: Status 404 returned error can't find the container with id 63941436dd22c376e05d022560844255ffc2c95835cc17b3638cb536dff0e6b1
Sep 11 15:56:00.201312 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:00.201294    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"]
Sep 11 15:56:00.203545 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:56:00.203525    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa70cb1_50f4_491d_a6b2_dd5b1372987f.slice/crio-6e9997e7dd413f5f0bc09bc16f7a9829017d14fd0225ce68d8ae0b5ae5d11d2c WatchSource:0}: Error finding container 6e9997e7dd413f5f0bc09bc16f7a9829017d14fd0225ce68d8ae0b5ae5d11d2c: Status 404 returned error can't find the container with id 6e9997e7dd413f5f0bc09bc16f7a9829017d14fd0225ce68d8ae0b5ae5d11d2c
Sep 11 15:56:01.062533 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:01.062494    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg" event={"ID":"be1b7ea3-3ed2-4a1d-909c-189b398f65c9","Type":"ContainerStarted","Data":"2922a3d220c7b7616803750b48af97d877136f9a2862a6aace6435b27cf41af5"}
Sep 11 15:56:01.064371 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:01.064335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v" event={"ID":"9a850363-389b-4e2f-8aa7-aaeaaca36488","Type":"ContainerStarted","Data":"63941436dd22c376e05d022560844255ffc2c95835cc17b3638cb536dff0e6b1"}
Sep 11 15:56:01.066532 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:01.066459    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g" event={"ID":"cfa70cb1-50f4-491d-a6b2-dd5b1372987f","Type":"ContainerStarted","Data":"6e9997e7dd413f5f0bc09bc16f7a9829017d14fd0225ce68d8ae0b5ae5d11d2c"}
Sep 11 15:56:03.170846 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:03.170753    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:03.325871 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:03.325807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:03.439225 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:03.439121    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:03.742701 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:03.742620    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:03.828398 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:03.828367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:04.483755 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:04.483718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:04.880940 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:04.880909    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:04.981386 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:04.981330    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:04.987836 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:04.987802    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4dp94"
Sep 11 15:56:05.669965 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:05.669925    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:05.735006 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:05.734962    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:06.068168 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:06.068138    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:06.122020 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:06.121941    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:06.369992 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:06.369970    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:06.473212 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:06.473191    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:06.682698 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:06.682619    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:07.094887 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.094850    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v" event={"ID":"9a850363-389b-4e2f-8aa7-aaeaaca36488","Type":"ContainerStarted","Data":"6e6d66c61199ba3088e911d26c2cb5cb66f573094bb116cd21d6dede60135d9d"}
Sep 11 15:56:07.095041 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.094897    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v" event={"ID":"9a850363-389b-4e2f-8aa7-aaeaaca36488","Type":"ContainerStarted","Data":"ec0a833455b27d19dc98605527930dead91da301ffec760c5fc2ade873d765cf"}
Sep 11 15:56:07.095041 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.094911    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v" event={"ID":"9a850363-389b-4e2f-8aa7-aaeaaca36488","Type":"ContainerStarted","Data":"a45cf8bbfcba41f135ea04b9646b50f3d0c6f69cd078a0da053b3f81b3ddae8a"}
Sep 11 15:56:07.096308 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.096280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g" event={"ID":"cfa70cb1-50f4-491d-a6b2-dd5b1372987f","Type":"ContainerStarted","Data":"d25d00bc25d250d9356b36c33234235229c34e0500eb664b3fc0278a6790f761"}
Sep 11 15:56:07.096457 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.096443    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:56:07.097697 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.097671    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg" event={"ID":"be1b7ea3-3ed2-4a1d-909c-189b398f65c9","Type":"ContainerStarted","Data":"154d467d8c8722b139d6e8a307a96e246d54cfd8011f4c3faf93e2feb604d2bb"}
Sep 11 15:56:07.098175 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.098159    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g"
Sep 11 15:56:07.113110 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.113071    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d44595964-zb47v" podStartSLOduration=1.9536404680000001 podStartE2EDuration="8.113059464s" totalImagesPullingTime="6.159418996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:07.111832759 +0000 UTC m=+60.829807879" watchObservedRunningTime="2026-09-11 15:56:07.113059464 +0000 UTC m=+60.831034582"
Sep 11 15:56:07.127500 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.127462    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-864bd58df6-wj5mg" podStartSLOduration=1.956867471 podStartE2EDuration="8.127446063s" totalImagesPullingTime="6.170578592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:07.126871362 +0000 UTC m=+60.844846482" watchObservedRunningTime="2026-09-11 15:56:07.127446063 +0000 UTC m=+60.845421184"
Sep 11 15:56:07.144295 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:07.144261    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-79cdcb5f84-4nz5g" podStartSLOduration=1.9881707990000002 podStartE2EDuration="8.144231288s" totalImagesPullingTime="6.156060489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:07.143681509 +0000 UTC m=+60.861656628" watchObservedRunningTime="2026-09-11 15:56:07.144231288 +0000 UTC m=+60.862206408"
Sep 11 15:56:08.644559 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:08.644525    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:08.748892 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:08.748859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:08.808679 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:08.808654    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:09.059554 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:09.059490    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-t48lx"
Sep 11 15:56:09.268396 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:09.268359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:09.769253 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:09.769216    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:09.871331 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:09.871296    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:10.108810 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:10.108780    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:10.205850 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:10.205816    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:10.306602 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:10.306575    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:10.407142 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:10.407073    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:10.764437 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:10.764370    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:10.828728 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:10.828701    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:11.167987 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.167966    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:11.228110 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.228089    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:11.517202 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.517139    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:11.618708 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.618684    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:11.730659 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.730632    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:11.830665 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.830630    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:11.835633 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.835614    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nszbh"]
Sep 11 15:56:11.850125 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.850104    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.853019 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.852988    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9j5tf\""
Sep 11 15:56:11.853323 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.853308    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 15:56:11.854260 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.854229    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 15:56:11.854374 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.854264    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 15:56:11.854374 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.854279    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 15:56:11.854374 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.854287    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 15:56:11.854374 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.854293    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 15:56:11.864872 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.864854    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-textfile\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.864955 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.864881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-metrics-client-ca\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.864955 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.864903    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-tls\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.864955 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.864936    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-wtmp\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.865053 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.864961    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-sys\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.865053 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.864989    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.865053 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.865004    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmpx\" (UniqueName: \"kubernetes.io/projected/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-kube-api-access-xnmpx\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.865053 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.865048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-root\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.865168 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.865065    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965315 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965295    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965415 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmpx\" (UniqueName: \"kubernetes.io/projected/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-kube-api-access-xnmpx\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965415 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-root\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965415 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965374    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965415 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-textfile\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965613 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965427    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-root\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965613 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-metrics-client-ca\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965613 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-tls\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965613 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965527    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-wtmp\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965613 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965554    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-sys\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965865 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:56:11.965615    2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 11 15:56:11.965865 ip-10-0-141-101 kubenswrapper[2564]: E0911 15:56:11.965674    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-tls podName:a5032b63-67a2-4f5f-8b1a-e491ac67a1a5 nodeName:}" failed. No retries permitted until 2026-09-11 15:56:12.465656892 +0000 UTC m=+66.183632005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-tls") pod "node-exporter-nszbh" (UID: "a5032b63-67a2-4f5f-8b1a-e491ac67a1a5") : secret "node-exporter-tls" not found
Sep 11 15:56:11.965865 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965616    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-sys\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965865 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965714    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-wtmp\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.965865 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-textfile\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.966025 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.965969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.966059 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.966027    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-metrics-client-ca\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.967636 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.967618    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:11.976099 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:11.976079    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmpx\" (UniqueName: \"kubernetes.io/projected/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-kube-api-access-xnmpx\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:12.013561 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.013522    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:12.113789 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.113732    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:12.214444 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.214413    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:12.314902 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.314882    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:12.442629 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.442570    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:12.469765 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.469741    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-tls\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:12.471839 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.471820    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a5032b63-67a2-4f5f-8b1a-e491ac67a1a5-node-exporter-tls\") pod \"node-exporter-nszbh\" (UID: \"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5\") " pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:12.570602 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.570578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:56:12.570683 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.570611    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:56:12.570683 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.570632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:56:12.573197 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.573180    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 15:56:12.573251 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.573180    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 15:56:12.573251 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.573211    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 15:56:12.583580 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.583558    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38668909-95f5-4bbb-84df-f71f4856578a-metrics-certs\") pod \"network-metrics-daemon-cfrmq\" (UID: \"38668909-95f5-4bbb-84df-f71f4856578a\") " pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:56:12.583676 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.583561    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8844b9e8-33aa-4253-8570-bf7b1b1c346e-original-pull-secret\") pod \"global-pull-secret-syncer-74rvc\" (UID: \"8844b9e8-33aa-4253-8570-bf7b1b1c346e\") " pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:56:12.583770 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.583756    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 15:56:12.594072 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.594054    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-585lg\" (UniqueName: \"kubernetes.io/projected/c0183175-4d09-407f-b523-00a608be58b8-kube-api-access-585lg\") pod \"network-check-target-k5h9f\" (UID: \"c0183175-4d09-407f-b523-00a608be58b8\") " pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:56:12.717185 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.717133    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-74rvc"
Sep 11 15:56:12.726665 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.726635    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lsspg\""
Sep 11 15:56:12.732067 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.732051    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zsbjs\""
Sep 11 15:56:12.735058 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.735044    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:56:12.740236 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.740224    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cfrmq"
Sep 11 15:56:12.742747 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.742720    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:12.759556 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.759539    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nszbh"
Sep 11 15:56:12.778944 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:56:12.778919    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5032b63_67a2_4f5f_8b1a_e491ac67a1a5.slice/crio-cdbc09ed9a597e8b69ae0aaf3dc25a2b3b2b2a09bee4a442ddc8130dce688401 WatchSource:0}: Error finding container cdbc09ed9a597e8b69ae0aaf3dc25a2b3b2b2a09bee4a442ddc8130dce688401: Status 404 returned error can't find the container with id cdbc09ed9a597e8b69ae0aaf3dc25a2b3b2b2a09bee4a442ddc8130dce688401
Sep 11 15:56:12.845017 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.844974    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:12.956055 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:12.956029    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:13.021662 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:13.021637    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-74rvc"]
Sep 11 15:56:13.023796 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:56:13.023771    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8844b9e8_33aa_4253_8570_bf7b1b1c346e.slice/crio-76ce8d476b0afce8d9f5a41e193469e595b12497512d7f95a7001b019aa40f6f WatchSource:0}: Error finding container 76ce8d476b0afce8d9f5a41e193469e595b12497512d7f95a7001b019aa40f6f: Status 404 returned error can't find the container with id 76ce8d476b0afce8d9f5a41e193469e595b12497512d7f95a7001b019aa40f6f
Sep 11 15:56:13.045706 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:13.045684    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k5h9f"]
Sep 11 15:56:13.048307 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:56:13.048284    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0183175_4d09_407f_b523_00a608be58b8.slice/crio-6ae3abb53759eb59f7b3cf7903d82cb5a3cc2adc317a09c158ae14b2668d1aa0 WatchSource:0}: Error finding container 6ae3abb53759eb59f7b3cf7903d82cb5a3cc2adc317a09c158ae14b2668d1aa0: Status 404 returned error can't find the container with id 6ae3abb53759eb59f7b3cf7903d82cb5a3cc2adc317a09c158ae14b2668d1aa0
Sep 11 15:56:13.079640 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:13.079621    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cfrmq"]
Sep 11 15:56:13.082116 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:56:13.082093    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38668909_95f5_4bbb_84df_f71f4856578a.slice/crio-6ee0ac7513ce370996e0aa3c70b4c96cfd5eddda97b64849935c2ae364ef477f WatchSource:0}: Error finding container 6ee0ac7513ce370996e0aa3c70b4c96cfd5eddda97b64849935c2ae364ef477f: Status 404 returned error can't find the container with id 6ee0ac7513ce370996e0aa3c70b4c96cfd5eddda97b64849935c2ae364ef477f
Sep 11 15:56:13.083334 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:13.083252    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:13.116763 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:13.116738    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nszbh" event={"ID":"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5","Type":"ContainerStarted","Data":"cdbc09ed9a597e8b69ae0aaf3dc25a2b3b2b2a09bee4a442ddc8130dce688401"}
Sep 11 15:56:13.117865 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:13.117844    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfrmq" event={"ID":"38668909-95f5-4bbb-84df-f71f4856578a","Type":"ContainerStarted","Data":"6ee0ac7513ce370996e0aa3c70b4c96cfd5eddda97b64849935c2ae364ef477f"}
Sep 11 15:56:13.118908 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:13.118889    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k5h9f" event={"ID":"c0183175-4d09-407f-b523-00a608be58b8","Type":"ContainerStarted","Data":"6ae3abb53759eb59f7b3cf7903d82cb5a3cc2adc317a09c158ae14b2668d1aa0"}
Sep 11 15:56:13.119891 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:13.119874    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-74rvc" event={"ID":"8844b9e8-33aa-4253-8570-bf7b1b1c346e","Type":"ContainerStarted","Data":"76ce8d476b0afce8d9f5a41e193469e595b12497512d7f95a7001b019aa40f6f"}
Sep 11 15:56:14.706961 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:14.706922    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:14.806189 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:14.806154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:15.081033 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.080999    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:15.130642 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.130607    2564 generic.go:356] "Generic (PLEG): container finished" podID="a5032b63-67a2-4f5f-8b1a-e491ac67a1a5" containerID="796d29d0cbb68ca0caf5fcaf62e7f0b0dfee2d7785605137c73a43c3f60578be" exitCode=0
Sep 11 15:56:15.130831 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.130700    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nszbh" event={"ID":"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5","Type":"ContainerDied","Data":"796d29d0cbb68ca0caf5fcaf62e7f0b0dfee2d7785605137c73a43c3f60578be"}
Sep 11 15:56:15.133377 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.133352    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfrmq" event={"ID":"38668909-95f5-4bbb-84df-f71f4856578a","Type":"ContainerStarted","Data":"c22e79c4a6f366bcee153b1ff8559f10316c3f0fdd68935c901e050327957f23"}
Sep 11 15:56:15.133489 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.133403    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cfrmq" event={"ID":"38668909-95f5-4bbb-84df-f71f4856578a","Type":"ContainerStarted","Data":"a2d8f679a61990644b989f86c1d1f40af07d09c89e7c44e8e4992110dd6e9ae8"}
Sep 11 15:56:15.170123 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.170078    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cfrmq" podStartSLOduration=66.784227039 podStartE2EDuration="1m8.170059481s" totalImagesPullingTime="1.385832442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:15.168916963 +0000 UTC m=+68.886892093" watchObservedRunningTime="2026-09-11 15:56:15.170059481 +0000 UTC m=+68.888034603"
Sep 11 15:56:15.269374 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.269322    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:15.370765 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.370692    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:15.548451 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.548414    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:15.649493 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:15.649422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:16.139892 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.139856    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nszbh" event={"ID":"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5","Type":"ContainerStarted","Data":"31b4d40ca0ef9fcd1b43793bd53cdbeac3f813800b6038e93f8793672ad5fd24"}
Sep 11 15:56:16.140307 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.139913    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nszbh" event={"ID":"a5032b63-67a2-4f5f-8b1a-e491ac67a1a5","Type":"ContainerStarted","Data":"883f652da912841bad21f6b35d4330a30190d936b920b0910b466312471e17f5"}
Sep 11 15:56:16.160996 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.160956    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nszbh" podStartSLOduration=3.390411933 podStartE2EDuration="5.16093809s" totalImagesPullingTime="1.685999627s" totalInitContainerRuntime="84.52653ms" isStatefulPod=true podCreationTimestamp="2026-09-11 15:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:16.159054274 +0000 UTC m=+69.877029424" watchObservedRunningTime="2026-09-11 15:56:16.16093809 +0000 UTC m=+69.878913210"
Sep 11 15:56:16.180770 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.180746    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"]
Sep 11 15:56:16.182538 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.182521    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.185689 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.185665    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 15:56:16.185689 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.185682    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 15:56:16.185834 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.185714    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 15:56:16.186470 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.186450    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-btalg60rcn5dg\""
Sep 11 15:56:16.186586 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.186514    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-tcdgs\""
Sep 11 15:56:16.186710 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.186695    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 15:56:16.195165 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.195145    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"]
Sep 11 15:56:16.195382 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.195365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6tc\" (UniqueName: \"kubernetes.io/projected/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-kube-api-access-gq6tc\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.195459 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.195415    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-secret-metrics-server-tls\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.195517 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.195487    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.195621 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.195520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-metrics-server-audit-profiles\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.195621 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.195547    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-client-ca-bundle\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.195967 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.195943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-secret-metrics-server-client-certs\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.196176 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.196152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-audit-log\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.296981 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.296933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-secret-metrics-server-client-certs\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.297184 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.296993    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-audit-log\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.297184 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.297020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6tc\" (UniqueName: \"kubernetes.io/projected/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-kube-api-access-gq6tc\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.297184 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.297046    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-secret-metrics-server-tls\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.297385 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.297231    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.297385 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.297301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-metrics-server-audit-profiles\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.297385 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.297327    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-client-ca-bundle\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.299662 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.297862    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-audit-log\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.299662 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.298306    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.299662 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.298522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-metrics-server-audit-profiles\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.300172 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.300145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-secret-metrics-server-client-certs\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.302334 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.301234    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-client-ca-bundle\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.302334 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.301347    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-secret-metrics-server-tls\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.306074 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.306049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6tc\" (UniqueName: \"kubernetes.io/projected/e1b7b75c-4b5b-424f-906d-3afba77b7e1a-kube-api-access-gq6tc\") pod \"metrics-server-57bcd8f66d-qcgg8\" (UID: \"e1b7b75c-4b5b-424f-906d-3afba77b7e1a\") " pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.493450 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.493359    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:16.532323 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.532286    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:16.612776 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:16.612740    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:17.084998 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:17.084948    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:17.944945 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:17.944868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"]
Sep 11 15:56:17.947860 ip-10-0-141-101 kubenswrapper[2564]: W0911 15:56:17.947832    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b7b75c_4b5b_424f_906d_3afba77b7e1a.slice/crio-24b6d249fb44cfb21c9c9b374f5ca44f63497c4c47caf15500b0dc70c481a1d6 WatchSource:0}: Error finding container 24b6d249fb44cfb21c9c9b374f5ca44f63497c4c47caf15500b0dc70c481a1d6: Status 404 returned error can't find the container with id 24b6d249fb44cfb21c9c9b374f5ca44f63497c4c47caf15500b0dc70c481a1d6
Sep 11 15:56:17.971709 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:17.971685    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:18.147363 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:18.147333    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k5h9f" event={"ID":"c0183175-4d09-407f-b523-00a608be58b8","Type":"ContainerStarted","Data":"9cf53eb08543a3e6bb6f81c6ad731085285f199b4c3cce30ebae05ca7eb184b8"}
Sep 11 15:56:18.147537 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:18.147521    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:56:18.148885 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:18.148859    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-74rvc" event={"ID":"8844b9e8-33aa-4253-8570-bf7b1b1c346e","Type":"ContainerStarted","Data":"0529983d5782f0debb142d4791ec8d4dda3d9e24a931f73cbf1e6d62256dd877"}
Sep 11 15:56:18.150024 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:18.149999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8" event={"ID":"e1b7b75c-4b5b-424f-906d-3afba77b7e1a","Type":"ContainerStarted","Data":"24b6d249fb44cfb21c9c9b374f5ca44f63497c4c47caf15500b0dc70c481a1d6"}
Sep 11 15:56:18.162794 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:18.162757    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k5h9f" podStartSLOduration=66.530394397 podStartE2EDuration="1m11.162745996s" totalImagesPullingTime="4.632351599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:18.161832478 +0000 UTC m=+71.879807739" watchObservedRunningTime="2026-09-11 15:56:18.162745996 +0000 UTC m=+71.880721115"
Sep 11 15:56:18.178855 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:18.178813    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-74rvc" podStartSLOduration=66.510443378 podStartE2EDuration="1m11.178802s" totalImagesPullingTime="4.668358622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 15:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:18.178321805 +0000 UTC m=+71.896296949" watchObservedRunningTime="2026-09-11 15:56:18.178802 +0000 UTC m=+71.896777148"
Sep 11 15:56:20.161091 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:20.161051    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8" event={"ID":"e1b7b75c-4b5b-424f-906d-3afba77b7e1a","Type":"ContainerStarted","Data":"599952dba78b8e19b643d9d0c8daa49405eb048ffb5246c449f5d5635ba174b0"}
Sep 11 15:56:20.180091 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:20.180044    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8" podStartSLOduration=2.8160569669999997 podStartE2EDuration="4.180028824s" totalImagesPullingTime="1.363971857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 15:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 15:56:20.17952762 +0000 UTC m=+73.897502761" watchObservedRunningTime="2026-09-11 15:56:20.180028824 +0000 UTC m=+73.898003943"
Sep 11 15:56:20.781036 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:20.781008    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:20.837763 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:20.837727    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:21.625649 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:21.625616    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:21.709193 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:21.709164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:21.971594 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:21.971506    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:22.031984 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:22.031959    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:24.600167 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:24.600133    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:24.675952 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:24.675918    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:24.762995 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:24.762968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:24.830486 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:24.830464    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:25.008415 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:25.008333    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:25.064775 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:25.064748    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:26.217623 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:26.217590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:26.318181 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:26.318145    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:28.405002 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:28.404956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:28.468018 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:28.467986    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:30.185183 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:30.185057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:30.281259 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:30.281217    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:33.822440 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:33.822402    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:33.923089 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:33.923056    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:34.820274 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:34.820223    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:34.882071 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:34.882037    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:35.293630 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:35.293555    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:35.394878 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:35.394840    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:36.494183 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:36.494150    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:36.494576 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:36.494253    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:38.256635 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:38.256605    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:38.358364 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:38.358332    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:39.016767 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:39.016736    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:39.374829 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:39.374792    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:39.434223 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:39.434193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:40.231287 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:40.231257    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:40.336280 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:40.336250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:49.158981 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:49.158954    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k5h9f"
Sep 11 15:56:54.761200 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:54.761161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:54.823996 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:54.823965    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:56:56.498854 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:56.498822    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:56:56.502404 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:56:56.502385    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57bcd8f66d-qcgg8"
Sep 11 15:57:31.001310 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:31.001232    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:57:31.305200 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:31.305103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:57:39.319925 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:39.319879    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:57:39.386539 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:39.386510    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:57:39.462889 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:39.462856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:57:40.048271 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:40.048224    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:57:40.150045 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:40.150019    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:57:50.338056 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:50.338019    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:57:50.402941 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:50.402908    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:57:50.470864 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:57:50.470826    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:18.941220 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:18.941120    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:19.128509 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:19.128471    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:32.893436 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:32.893400    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:46.321557 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:46.321515    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:46.402102 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:46.402064    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:49.605499 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:49.605469    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:49.706691 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:49.706649    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:49.776704 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:49.776675    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:50.329993 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:50.329962    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:50.409702 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:50.409678    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:58.375956 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:58.375906    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:58:58.479060 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:58:58.479020    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:59:00.671641 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:59:00.671610    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:59:00.787005 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:59:00.786974    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:59:00.921347 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:59:00.921321    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:59:09.785401 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:59:09.785362    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:59:09.892687 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:59:09.892624    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:59:09.970083 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:59:09.970047    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 15:59:41.749383 ip-10-0-141-101 kubenswrapper[2564]: I0911 15:59:41.749351    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:02.619043 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:02.618997    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:05.962865 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:05.962811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:06.690863 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:06.690833    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 16:00:22.456621 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:22.456575    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:29.708394 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:29.708354    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:30.533514 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:30.533470    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:33.546699 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:33.546660    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:33.644426 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:33.644386    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:35.281985 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:35.281946    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:35.426313 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:35.426273    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:35.747261 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:35.747209    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:35.838800 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:35.838773    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:38.120015 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:38.119974    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:38.287962 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:38.287931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:40.319166 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:40.319126    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:40.398129 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:40.398100    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:42.764082 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.764021    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"]
Sep 11 16:00:42.766123 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.766103    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"
Sep 11 16:00:42.768714 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.768679    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 16:00:42.768987 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.768955    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 16:00:42.769957 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.769928    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2k64w\""
Sep 11 16:00:42.775233 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.775197    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"]
Sep 11 16:00:42.786854 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.786800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2015ad5-8391-4ba7-9265-1742a54693b8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m77dq\" (UID: \"b2015ad5-8391-4ba7-9265-1742a54693b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"
Sep 11 16:00:42.786954 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.786867    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dpwm\" (UniqueName: \"kubernetes.io/projected/b2015ad5-8391-4ba7-9265-1742a54693b8-kube-api-access-7dpwm\") pod \"cert-manager-cainjector-559cdd8c87-m77dq\" (UID: \"b2015ad5-8391-4ba7-9265-1742a54693b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"
Sep 11 16:00:42.887595 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.887564    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2015ad5-8391-4ba7-9265-1742a54693b8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m77dq\" (UID: \"b2015ad5-8391-4ba7-9265-1742a54693b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"
Sep 11 16:00:42.887726 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.887613    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dpwm\" (UniqueName: \"kubernetes.io/projected/b2015ad5-8391-4ba7-9265-1742a54693b8-kube-api-access-7dpwm\") pod \"cert-manager-cainjector-559cdd8c87-m77dq\" (UID: \"b2015ad5-8391-4ba7-9265-1742a54693b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"
Sep 11 16:00:42.897005 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.896976    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b2015ad5-8391-4ba7-9265-1742a54693b8-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m77dq\" (UID: \"b2015ad5-8391-4ba7-9265-1742a54693b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"
Sep 11 16:00:42.897131 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:42.897101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dpwm\" (UniqueName: \"kubernetes.io/projected/b2015ad5-8391-4ba7-9265-1742a54693b8-kube-api-access-7dpwm\") pod \"cert-manager-cainjector-559cdd8c87-m77dq\" (UID: \"b2015ad5-8391-4ba7-9265-1742a54693b8\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"
Sep 11 16:00:43.076745 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:43.076719    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"
Sep 11 16:00:43.349351 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:43.349282    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m77dq"]
Sep 11 16:00:43.353336 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:00:43.353306    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2015ad5_8391_4ba7_9265_1742a54693b8.slice/crio-cc914cefdd9295b02f03b58d9b5169f71d9ae507a5dcc49f1c7887aba56a0db3 WatchSource:0}: Error finding container cc914cefdd9295b02f03b58d9b5169f71d9ae507a5dcc49f1c7887aba56a0db3: Status 404 returned error can't find the container with id cc914cefdd9295b02f03b58d9b5169f71d9ae507a5dcc49f1c7887aba56a0db3
Sep 11 16:00:43.355332 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:43.355312    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:00:44.004965 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:44.004886    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq" event={"ID":"b2015ad5-8391-4ba7-9265-1742a54693b8","Type":"ContainerStarted","Data":"cc914cefdd9295b02f03b58d9b5169f71d9ae507a5dcc49f1c7887aba56a0db3"}
Sep 11 16:00:45.692439 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:45.692383    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:46.889106 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:46.889069    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:46.970110 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:46.970081    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:47.016923 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:47.016895    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq" event={"ID":"b2015ad5-8391-4ba7-9265-1742a54693b8","Type":"ContainerStarted","Data":"3830067e01c5bff674d1db596ab13f319004c73b6c4d32b2ed4ca5ae0c969226"}
Sep 11 16:00:47.043701 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:47.043661    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m77dq" podStartSLOduration=2.286109431 podStartE2EDuration="5.04364654s" totalImagesPullingTime="2.757537109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:00:47.042906877 +0000 UTC m=+340.760881998" watchObservedRunningTime="2026-09-11 16:00:47.04364654 +0000 UTC m=+340.761621660"
Sep 11 16:00:47.780358 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:47.780313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:47.881140 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:47.881111    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:54.412015 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:54.411968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:54.522504 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:54.522473    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:00:56.196604 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:00:56.196562    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:10.758864 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:10.758775    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:10.965925 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:10.965888    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:11.104404 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:11.104374    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:11.231390 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:11.231364    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:11.509644 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:11.509568    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:12.045740 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.045705    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:12.154537 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.154489    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:12.198623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.198598    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"]
Sep 11 16:01:12.201770 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.201753    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.204599 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.204576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wl6q7\""
Sep 11 16:01:12.204991 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.204977    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 16:01:12.206332 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.206315    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 16:01:12.206803 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.206787    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 16:01:12.206861 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.206843    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 16:01:12.227434 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.227412    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"]
Sep 11 16:01:12.272925 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.272902    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8014c0da-410d-409e-8d5f-0bbf23f89b24-manifests\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.273017 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.272940    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8014c0da-410d-409e-8d5f-0bbf23f89b24-webhook-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.273017 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.272966    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8014c0da-410d-409e-8d5f-0bbf23f89b24-tmp\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.273101 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.273018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8014c0da-410d-409e-8d5f-0bbf23f89b24-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.273101 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.273040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcgs\" (UniqueName: \"kubernetes.io/projected/8014c0da-410d-409e-8d5f-0bbf23f89b24-kube-api-access-mhcgs\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.373567 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.373544    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8014c0da-410d-409e-8d5f-0bbf23f89b24-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.373678 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.373571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcgs\" (UniqueName: \"kubernetes.io/projected/8014c0da-410d-409e-8d5f-0bbf23f89b24-kube-api-access-mhcgs\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.373678 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.373598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8014c0da-410d-409e-8d5f-0bbf23f89b24-manifests\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.373678 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.373624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8014c0da-410d-409e-8d5f-0bbf23f89b24-webhook-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.373678 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.373642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8014c0da-410d-409e-8d5f-0bbf23f89b24-tmp\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.373984 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.373961    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8014c0da-410d-409e-8d5f-0bbf23f89b24-manifests\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.376280 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.376233    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8014c0da-410d-409e-8d5f-0bbf23f89b24-tmp\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.376425 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.376407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8014c0da-410d-409e-8d5f-0bbf23f89b24-webhook-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.376506 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.376487    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8014c0da-410d-409e-8d5f-0bbf23f89b24-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.385176 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.385153    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcgs\" (UniqueName: \"kubernetes.io/projected/8014c0da-410d-409e-8d5f-0bbf23f89b24-kube-api-access-mhcgs\") pod \"opendatahub-operator-controller-manager-58647b66f4-lqgls\" (UID: \"8014c0da-410d-409e-8d5f-0bbf23f89b24\") " pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.438960 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.438933    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:12.512564 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.512541    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:12.539301 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.539274    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:12.637750 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.637670    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:12.740490 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.740463    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:12.789525 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.789502    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"]
Sep 11 16:01:12.792006 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:01:12.791984    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8014c0da_410d_409e_8d5f_0bbf23f89b24.slice/crio-30cb937989e7b5afa075bbb731cb00cce7a49e15d70c147699ee5a8001dd9b94 WatchSource:0}: Error finding container 30cb937989e7b5afa075bbb731cb00cce7a49e15d70c147699ee5a8001dd9b94: Status 404 returned error can't find the container with id 30cb937989e7b5afa075bbb731cb00cce7a49e15d70c147699ee5a8001dd9b94
Sep 11 16:01:12.818058 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:12.818033    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:13.104140 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:13.104110    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls" event={"ID":"8014c0da-410d-409e-8d5f-0bbf23f89b24","Type":"ContainerStarted","Data":"30cb937989e7b5afa075bbb731cb00cce7a49e15d70c147699ee5a8001dd9b94"}
Sep 11 16:01:16.117096 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:16.117066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls" event={"ID":"8014c0da-410d-409e-8d5f-0bbf23f89b24","Type":"ContainerStarted","Data":"f80f9f011a6802bcb60979a9ceb145ef0c72242f4d82424aaeabd863edae9339"}
Sep 11 16:01:17.123204 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:17.123132    2564 generic.go:356] "Generic (PLEG): container finished" podID="8014c0da-410d-409e-8d5f-0bbf23f89b24" containerID="f80f9f011a6802bcb60979a9ceb145ef0c72242f4d82424aaeabd863edae9339" exitCode=0
Sep 11 16:01:17.123545 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:17.123208    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls" event={"ID":"8014c0da-410d-409e-8d5f-0bbf23f89b24","Type":"ContainerDied","Data":"f80f9f011a6802bcb60979a9ceb145ef0c72242f4d82424aaeabd863edae9339"}
Sep 11 16:01:18.128353 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:18.128318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls" event={"ID":"8014c0da-410d-409e-8d5f-0bbf23f89b24","Type":"ContainerStarted","Data":"87c432c7763d7a909b4af048e1f7a52c8390639a10284b3c053b07283642df6e"}
Sep 11 16:01:18.128728 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:18.128407    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:18.152189 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:18.152132    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls" podStartSLOduration=1.791198535 podStartE2EDuration="6.152112628s" totalImagesPullingTime="3.346606578s" totalInitContainerRuntime="1.014307515s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:18.151535461 +0000 UTC m=+371.869510596" watchObservedRunningTime="2026-09-11 16:01:18.152112628 +0000 UTC m=+371.870087740"
Sep 11 16:01:25.332733 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:25.332685    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:25.397705 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:25.397671    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:25.472380 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:25.472349    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:25.534530 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:25.534496    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:26.067780 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:26.067750    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:26.127371 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:26.127336    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:29.135124 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:29.135091    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58647b66f4-lqgls"
Sep 11 16:01:31.596476 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:31.596442    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:31.733739 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:31.733708    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:32.234976 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.234939    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:32.271133 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.271107    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"]
Sep 11 16:01:32.274136 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.274121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.277529 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.277510    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 16:01:32.277645 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.277628    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 16:01:32.278777 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.278760    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 16:01:32.278853 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.278811    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-42btk\""
Sep 11 16:01:32.281624 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.281609    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 16:01:32.286935 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.286910    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"]
Sep 11 16:01:32.305436 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.305418    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95b3495f-b1d2-4d9e-ab6e-0db4d623dd97-tmp\") pod \"kube-auth-proxy-6c65944c75-l7sqc\" (UID: \"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.305523 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.305455    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b3495f-b1d2-4d9e-ab6e-0db4d623dd97-tls-certs\") pod \"kube-auth-proxy-6c65944c75-l7sqc\" (UID: \"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.305523 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.305473    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8zxs\" (UniqueName: \"kubernetes.io/projected/95b3495f-b1d2-4d9e-ab6e-0db4d623dd97-kube-api-access-s8zxs\") pod \"kube-auth-proxy-6c65944c75-l7sqc\" (UID: \"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.367338 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.367316    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:32.406435 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.406410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b3495f-b1d2-4d9e-ab6e-0db4d623dd97-tls-certs\") pod \"kube-auth-proxy-6c65944c75-l7sqc\" (UID: \"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.406529 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.406442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8zxs\" (UniqueName: \"kubernetes.io/projected/95b3495f-b1d2-4d9e-ab6e-0db4d623dd97-kube-api-access-s8zxs\") pod \"kube-auth-proxy-6c65944c75-l7sqc\" (UID: \"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.406529 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.406504    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95b3495f-b1d2-4d9e-ab6e-0db4d623dd97-tmp\") pod \"kube-auth-proxy-6c65944c75-l7sqc\" (UID: \"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.408657 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.408638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/95b3495f-b1d2-4d9e-ab6e-0db4d623dd97-tmp\") pod \"kube-auth-proxy-6c65944c75-l7sqc\" (UID: \"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.408810 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.408793    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95b3495f-b1d2-4d9e-ab6e-0db4d623dd97-tls-certs\") pod \"kube-auth-proxy-6c65944c75-l7sqc\" (UID: \"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.416167 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.416145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8zxs\" (UniqueName: \"kubernetes.io/projected/95b3495f-b1d2-4d9e-ab6e-0db4d623dd97-kube-api-access-s8zxs\") pod \"kube-auth-proxy-6c65944c75-l7sqc\" (UID: \"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97\") " pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.583092 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.583069    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"
Sep 11 16:01:32.859149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.859120    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc"]
Sep 11 16:01:32.859648 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:01:32.859627    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b3495f_b1d2_4d9e_ab6e_0db4d623dd97.slice/crio-664f88c7c0397a552f6c22319534d1c098fab3fb171c0d327424165654a587df WatchSource:0}: Error finding container 664f88c7c0397a552f6c22319534d1c098fab3fb171c0d327424165654a587df: Status 404 returned error can't find the container with id 664f88c7c0397a552f6c22319534d1c098fab3fb171c0d327424165654a587df
Sep 11 16:01:32.925879 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:32.925845    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:33.187107 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.187046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc" event={"ID":"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97","Type":"ContainerStarted","Data":"664f88c7c0397a552f6c22319534d1c098fab3fb171c0d327424165654a587df"}
Sep 11 16:01:33.931477 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.931444    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"]
Sep 11 16:01:33.934537 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.934518    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:33.938256 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.938214    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 16:01:33.943577 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.943551    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dc27x\""
Sep 11 16:01:33.945712 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.945683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 16:01:33.949130 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.949112    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 16:01:33.949380 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.949365    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 16:01:33.977439 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.977375    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"]
Sep 11 16:01:33.979766 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:33.979742    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:34.018824 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.018798    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc68r\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-kube-api-access-bc68r\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.018956 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.018840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.018956 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.018871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.019080 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.018956    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.019080 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.019020    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28485785-8dc5-41a6-87b7-ae60dd8d3951-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.019080 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.019050    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.019232 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.019081    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.112218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.112186    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:34.120013 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.119988    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28485785-8dc5-41a6-87b7-ae60dd8d3951-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.120137 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.120024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.120137 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.120053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.120137 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.120095    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc68r\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-kube-api-access-bc68r\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.120137 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.120125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.120367 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.120149    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.120367 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.120195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.120841 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.120820    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.122907 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.122886    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28485785-8dc5-41a6-87b7-ae60dd8d3951-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.123021 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.122900    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.123021 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.122969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.123098 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.123070    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.129752 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.129728    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.130271 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.130222    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc68r\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-kube-api-access-bc68r\") pod \"istiod-openshift-gateway-7b7995d5cd-n244c\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:34.245042 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:34.244769    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:35.134479 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:35.134359    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"]
Sep 11 16:01:35.322596 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:35.322542    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:35.661370 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:01:35.661291    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28485785_8dc5_41a6_87b7_ae60dd8d3951.slice/crio-861d18fab9eba4c2a86d2962b70a19b193e0dd54d7ff64ba17b767107a3ef1f5 WatchSource:0}: Error finding container 861d18fab9eba4c2a86d2962b70a19b193e0dd54d7ff64ba17b767107a3ef1f5: Status 404 returned error can't find the container with id 861d18fab9eba4c2a86d2962b70a19b193e0dd54d7ff64ba17b767107a3ef1f5
Sep 11 16:01:36.201407 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:36.201368    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc" event={"ID":"95b3495f-b1d2-4d9e-ab6e-0db4d623dd97","Type":"ContainerStarted","Data":"871c12d60763f37b8779b2d3d14e75b4b8a89b2134f30497adec44d1f91288c5"}
Sep 11 16:01:36.202606 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:36.202580    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c" event={"ID":"28485785-8dc5-41a6-87b7-ae60dd8d3951","Type":"ContainerStarted","Data":"861d18fab9eba4c2a86d2962b70a19b193e0dd54d7ff64ba17b767107a3ef1f5"}
Sep 11 16:01:36.219134 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:36.219089    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c65944c75-l7sqc" podStartSLOduration=1.3622169419999999 podStartE2EDuration="4.219074397s" totalImagesPullingTime="2.856857455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:36.219030318 +0000 UTC m=+389.937005464" watchObservedRunningTime="2026-09-11 16:01:36.219074397 +0000 UTC m=+389.937049516"
Sep 11 16:01:36.450658 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:36.450619    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:36.551080 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:36.551010    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:38.110533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:38.110498    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 16:01:38.110886 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:38.110574    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 16:01:38.212000 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:38.211972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c" event={"ID":"28485785-8dc5-41a6-87b7-ae60dd8d3951","Type":"ContainerStarted","Data":"595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4"}
Sep 11 16:01:38.212106 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:38.212042    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:38.238971 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:38.238930    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c" podStartSLOduration=2.792398412 podStartE2EDuration="5.238915345s" totalImagesPullingTime="2.446516933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:38.238443144 +0000 UTC m=+391.956418281" watchObservedRunningTime="2026-09-11 16:01:38.238915345 +0000 UTC m=+391.956890465"
Sep 11 16:01:38.980151 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:38.980110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:39.083877 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:39.083839    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:39.217305 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:39.217270    2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-n244c container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 16:01:39.217639 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:39.217333    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c" podUID="28485785-8dc5-41a6-87b7-ae60dd8d3951" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 16:01:39.701782 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:39.701752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:39.952713 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:39.952636    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:40.064507 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:40.064475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:42.216929 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.216900    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:42.675713 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.675673    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wgdnm"]
Sep 11 16:01:42.680208 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.680189    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:42.682996 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.682970    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 11 16:01:42.683122 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.683031    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rzkhr\""
Sep 11 16:01:42.688632 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.688081    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wgdnm"]
Sep 11 16:01:42.739720 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.739694    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:42.785099 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.785069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0dad96b-caaf-4841-99e4-82ceecef8dfb-cert\") pod \"odh-model-controller-7998c67dc4-wgdnm\" (UID: \"d0dad96b-caaf-4841-99e4-82ceecef8dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:42.785231 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.785121    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v87vz\" (UniqueName: \"kubernetes.io/projected/d0dad96b-caaf-4841-99e4-82ceecef8dfb-kube-api-access-v87vz\") pod \"odh-model-controller-7998c67dc4-wgdnm\" (UID: \"d0dad96b-caaf-4841-99e4-82ceecef8dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:42.841206 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.841170    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:42.885629 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.885588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0dad96b-caaf-4841-99e4-82ceecef8dfb-cert\") pod \"odh-model-controller-7998c67dc4-wgdnm\" (UID: \"d0dad96b-caaf-4841-99e4-82ceecef8dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:42.885782 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.885652    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87vz\" (UniqueName: \"kubernetes.io/projected/d0dad96b-caaf-4841-99e4-82ceecef8dfb-kube-api-access-v87vz\") pod \"odh-model-controller-7998c67dc4-wgdnm\" (UID: \"d0dad96b-caaf-4841-99e4-82ceecef8dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:42.885782 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:01:42.885736    2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 11 16:01:42.885889 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:01:42.885805    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0dad96b-caaf-4841-99e4-82ceecef8dfb-cert podName:d0dad96b-caaf-4841-99e4-82ceecef8dfb nodeName:}" failed. No retries permitted until 2026-09-11 16:01:43.385788858 +0000 UTC m=+397.103763956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d0dad96b-caaf-4841-99e4-82ceecef8dfb-cert") pod "odh-model-controller-7998c67dc4-wgdnm" (UID: "d0dad96b-caaf-4841-99e4-82ceecef8dfb") : secret "odh-model-controller-webhook-cert" not found
Sep 11 16:01:42.897801 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.897767    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87vz\" (UniqueName: \"kubernetes.io/projected/d0dad96b-caaf-4841-99e4-82ceecef8dfb-kube-api-access-v87vz\") pod \"odh-model-controller-7998c67dc4-wgdnm\" (UID: \"d0dad96b-caaf-4841-99e4-82ceecef8dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:42.946815 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:42.946749    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:43.110537 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:43.110506    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:43.298106 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:43.298017    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:43.390418 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:43.390388    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0dad96b-caaf-4841-99e4-82ceecef8dfb-cert\") pod \"odh-model-controller-7998c67dc4-wgdnm\" (UID: \"d0dad96b-caaf-4841-99e4-82ceecef8dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:43.392942 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:43.392914    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0dad96b-caaf-4841-99e4-82ceecef8dfb-cert\") pod \"odh-model-controller-7998c67dc4-wgdnm\" (UID: \"d0dad96b-caaf-4841-99e4-82ceecef8dfb\") " pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:43.398722 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:43.398688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:43.592169 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:43.592127    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:43.859203 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:43.859114    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wgdnm"]
Sep 11 16:01:43.861840 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:01:43.861811    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0dad96b_caaf_4841_99e4_82ceecef8dfb.slice/crio-be3fbca3b21ffbe9446397b97231483296ded6eeee3ee7863141f6112e097dab WatchSource:0}: Error finding container be3fbca3b21ffbe9446397b97231483296ded6eeee3ee7863141f6112e097dab: Status 404 returned error can't find the container with id be3fbca3b21ffbe9446397b97231483296ded6eeee3ee7863141f6112e097dab
Sep 11 16:01:44.235227 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:44.235147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm" event={"ID":"d0dad96b-caaf-4841-99e4-82ceecef8dfb","Type":"ContainerStarted","Data":"be3fbca3b21ffbe9446397b97231483296ded6eeee3ee7863141f6112e097dab"}
Sep 11 16:01:47.252100 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:47.252056    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm" event={"ID":"d0dad96b-caaf-4841-99e4-82ceecef8dfb","Type":"ContainerStarted","Data":"c771166ac95d803e0272e0b93f1b5d389f2c271a2fb156a447cf96eb6fb82051"}
Sep 11 16:01:47.252574 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:47.252171    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:01:47.277376 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:47.277327    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm" podStartSLOduration=2.593283684 podStartE2EDuration="5.277307922s" totalImagesPullingTime="2.684024238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:47.274655147 +0000 UTC m=+400.992630285" watchObservedRunningTime="2026-09-11 16:01:47.277307922 +0000 UTC m=+400.995283043"
Sep 11 16:01:49.659795 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.659763    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"]
Sep 11 16:01:49.733488 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.733457    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"]
Sep 11 16:01:49.733630 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.733546    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.841681 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.841656    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.841681 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.841681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.841843 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.841700    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dee3238e-1a57-4905-a3f7-fbaf04baa517-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.841843 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.841732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.841843 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.841769    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.841843 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.841789    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee3238e-1a57-4905-a3f7-fbaf04baa517-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.841982 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.841842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pt77\" (UniqueName: \"kubernetes.io/projected/dee3238e-1a57-4905-a3f7-fbaf04baa517-kube-api-access-5pt77\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.942410 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.942350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dee3238e-1a57-4905-a3f7-fbaf04baa517-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.942410 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.942403    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.942605 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.942425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.942605 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.942451    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee3238e-1a57-4905-a3f7-fbaf04baa517-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.942605 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.942500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pt77\" (UniqueName: \"kubernetes.io/projected/dee3238e-1a57-4905-a3f7-fbaf04baa517-kube-api-access-5pt77\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.942605 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.942543    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.942605 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.942566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.943257 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.943215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.944805 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.944780    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dee3238e-1a57-4905-a3f7-fbaf04baa517-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.944917 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.944839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dee3238e-1a57-4905-a3f7-fbaf04baa517-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.945087 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.945071    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.945200 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.945180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.951362 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.951338    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dee3238e-1a57-4905-a3f7-fbaf04baa517-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:49.951764 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:49.951747    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pt77\" (UniqueName: \"kubernetes.io/projected/dee3238e-1a57-4905-a3f7-fbaf04baa517-kube-api-access-5pt77\") pod \"istiod-openshift-gateway-5677ccb49b-7zbkf\" (UID: \"dee3238e-1a57-4905-a3f7-fbaf04baa517\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:50.042332 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:50.042310    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:50.342087 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:50.342054    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"]
Sep 11 16:01:50.345089 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:01:50.345061    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee3238e_1a57_4905_a3f7_fbaf04baa517.slice/crio-7e8be3d448845d2b29d45b4531cc5066dc8eba61ca695505854bbc6ada71106a WatchSource:0}: Error finding container 7e8be3d448845d2b29d45b4531cc5066dc8eba61ca695505854bbc6ada71106a: Status 404 returned error can't find the container with id 7e8be3d448845d2b29d45b4531cc5066dc8eba61ca695505854bbc6ada71106a
Sep 11 16:01:50.348500 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:50.348471    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 16:01:50.348603 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:50.348529    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 11 16:01:51.269376 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.269334    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf" event={"ID":"dee3238e-1a57-4905-a3f7-fbaf04baa517","Type":"ContainerStarted","Data":"d26a92a9941342085f92c81612279066800b2d097d2ed72f2f2f4f91d4bfbed0"}
Sep 11 16:01:51.269376 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.269383    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf" event={"ID":"dee3238e-1a57-4905-a3f7-fbaf04baa517","Type":"ContainerStarted","Data":"7e8be3d448845d2b29d45b4531cc5066dc8eba61ca695505854bbc6ada71106a"}
Sep 11 16:01:51.269881 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.269468    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:51.271091 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.271069    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf"
Sep 11 16:01:51.303072 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.303021    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-7zbkf" podStartSLOduration=2.303006774 podStartE2EDuration="2.303006774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:51.300995944 +0000 UTC m=+405.018971062" watchObservedRunningTime="2026-09-11 16:01:51.303006774 +0000 UTC m=+405.020981893"
Sep 11 16:01:51.394711 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.394669    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"]
Sep 11 16:01:51.395165 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.395096    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c" podUID="28485785-8dc5-41a6-87b7-ae60dd8d3951" containerName="discovery" containerID="cri-o://595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4" gracePeriod=30
Sep 11 16:01:51.626845 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.626822    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:51.757679 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.757653    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-dns-cert\") pod \"28485785-8dc5-41a6-87b7-ae60dd8d3951\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") "
Sep 11 16:01:51.757796 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.757685    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-cacerts\") pod \"28485785-8dc5-41a6-87b7-ae60dd8d3951\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") "
Sep 11 16:01:51.757796 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.757721    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc68r\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-kube-api-access-bc68r\") pod \"28485785-8dc5-41a6-87b7-ae60dd8d3951\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") "
Sep 11 16:01:51.757796 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.757771    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-token\") pod \"28485785-8dc5-41a6-87b7-ae60dd8d3951\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") "
Sep 11 16:01:51.757901 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.757804    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28485785-8dc5-41a6-87b7-ae60dd8d3951-local-certs\") pod \"28485785-8dc5-41a6-87b7-ae60dd8d3951\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") "
Sep 11 16:01:51.757901 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.757854    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-ca-configmap\") pod \"28485785-8dc5-41a6-87b7-ae60dd8d3951\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") "
Sep 11 16:01:51.757901 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.757884    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-kubeconfig\") pod \"28485785-8dc5-41a6-87b7-ae60dd8d3951\" (UID: \"28485785-8dc5-41a6-87b7-ae60dd8d3951\") "
Sep 11 16:01:51.758310 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.758277    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "28485785-8dc5-41a6-87b7-ae60dd8d3951" (UID: "28485785-8dc5-41a6-87b7-ae60dd8d3951"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:01:51.760277 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.760203    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-kube-api-access-bc68r" (OuterVolumeSpecName: "kube-api-access-bc68r") pod "28485785-8dc5-41a6-87b7-ae60dd8d3951" (UID: "28485785-8dc5-41a6-87b7-ae60dd8d3951"). InnerVolumeSpecName "kube-api-access-bc68r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:01:51.760277 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.760228    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-cacerts" (OuterVolumeSpecName: "cacerts") pod "28485785-8dc5-41a6-87b7-ae60dd8d3951" (UID: "28485785-8dc5-41a6-87b7-ae60dd8d3951"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:01:51.760547 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.760287    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28485785-8dc5-41a6-87b7-ae60dd8d3951-local-certs" (OuterVolumeSpecName: "local-certs") pod "28485785-8dc5-41a6-87b7-ae60dd8d3951" (UID: "28485785-8dc5-41a6-87b7-ae60dd8d3951"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:01:51.760547 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.760310    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "28485785-8dc5-41a6-87b7-ae60dd8d3951" (UID: "28485785-8dc5-41a6-87b7-ae60dd8d3951"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:01:51.760547 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.760374    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-token" (OuterVolumeSpecName: "istio-token") pod "28485785-8dc5-41a6-87b7-ae60dd8d3951" (UID: "28485785-8dc5-41a6-87b7-ae60dd8d3951"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:01:51.760547 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.760376    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "28485785-8dc5-41a6-87b7-ae60dd8d3951" (UID: "28485785-8dc5-41a6-87b7-ae60dd8d3951"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:01:51.859358 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.859332    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-token\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:01:51.859358 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.859356    2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28485785-8dc5-41a6-87b7-ae60dd8d3951-local-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:01:51.859507 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.859370    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-ca-configmap\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:01:51.859507 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.859382    2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-kubeconfig\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:01:51.859507 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.859394    2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-istio-csr-dns-cert\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:01:51.859507 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.859406    2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28485785-8dc5-41a6-87b7-ae60dd8d3951-cacerts\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:01:51.859507 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:51.859417    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc68r\" (UniqueName: \"kubernetes.io/projected/28485785-8dc5-41a6-87b7-ae60dd8d3951-kube-api-access-bc68r\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:01:52.274397 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.274312    2564 generic.go:356] "Generic (PLEG): container finished" podID="28485785-8dc5-41a6-87b7-ae60dd8d3951" containerID="595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4" exitCode=0
Sep 11 16:01:52.274397 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.274355    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"
Sep 11 16:01:52.274851 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.274399    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c" event={"ID":"28485785-8dc5-41a6-87b7-ae60dd8d3951","Type":"ContainerDied","Data":"595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4"}
Sep 11 16:01:52.274851 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.274452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c" event={"ID":"28485785-8dc5-41a6-87b7-ae60dd8d3951","Type":"ContainerDied","Data":"861d18fab9eba4c2a86d2962b70a19b193e0dd54d7ff64ba17b767107a3ef1f5"}
Sep 11 16:01:52.274851 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.274480    2564 scope.go:117] "RemoveContainer" containerID="595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4"
Sep 11 16:01:52.283050 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.283033    2564 scope.go:117] "RemoveContainer" containerID="595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4"
Sep 11 16:01:52.283339 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:01:52.283319    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4\": container with ID starting with 595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4 not found: ID does not exist" containerID="595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4"
Sep 11 16:01:52.283397 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.283351    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4"} err="failed to get container status \"595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4\": rpc error: code = NotFound desc = could not find container \"595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4\": container with ID starting with 595e05c7ba491045d0a486e5cc5dd72bec4d2cde6281cb32bb9f97733ffd8ad4 not found: ID does not exist"
Sep 11 16:01:52.305339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.305317    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"]
Sep 11 16:01:52.306901 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.306882    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-n244c"]
Sep 11 16:01:52.795856 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.795807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:52.811292 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.811263    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28485785-8dc5-41a6-87b7-ae60dd8d3951" path="/var/lib/kubelet/pods/28485785-8dc5-41a6-87b7-ae60dd8d3951/volumes"
Sep 11 16:01:52.858618 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.858591    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"]
Sep 11 16:01:52.858841 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.858830    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28485785-8dc5-41a6-87b7-ae60dd8d3951" containerName="discovery"
Sep 11 16:01:52.858886 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.858843    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="28485785-8dc5-41a6-87b7-ae60dd8d3951" containerName="discovery"
Sep 11 16:01:52.858918 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.858886    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="28485785-8dc5-41a6-87b7-ae60dd8d3951" containerName="discovery"
Sep 11 16:01:52.862295 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.862279    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:52.865547 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.865529    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 16:01:52.865832 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.865815    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-bnv8s\""
Sep 11 16:01:52.875063 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.875040    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"]
Sep 11 16:01:52.906428 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.906407    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:52.966678 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.966651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8l9l\" (UniqueName: \"kubernetes.io/projected/0bc70645-7134-43fd-9294-da3a3651d464-kube-api-access-d8l9l\") pod \"kserve-controller-manager-d6f4b65b8-hwz4v\" (UID: \"0bc70645-7134-43fd-9294-da3a3651d464\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:52.966787 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.966682    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bc70645-7134-43fd-9294-da3a3651d464-cert\") pod \"kserve-controller-manager-d6f4b65b8-hwz4v\" (UID: \"0bc70645-7134-43fd-9294-da3a3651d464\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:52.999573 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:52.999549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:53.067030 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:53.067002    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8l9l\" (UniqueName: \"kubernetes.io/projected/0bc70645-7134-43fd-9294-da3a3651d464-kube-api-access-d8l9l\") pod \"kserve-controller-manager-d6f4b65b8-hwz4v\" (UID: \"0bc70645-7134-43fd-9294-da3a3651d464\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:53.067132 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:53.067036    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bc70645-7134-43fd-9294-da3a3651d464-cert\") pod \"kserve-controller-manager-d6f4b65b8-hwz4v\" (UID: \"0bc70645-7134-43fd-9294-da3a3651d464\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:53.067173 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:01:53.067153    2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 11 16:01:53.067227 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:01:53.067217    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bc70645-7134-43fd-9294-da3a3651d464-cert podName:0bc70645-7134-43fd-9294-da3a3651d464 nodeName:}" failed. No retries permitted until 2026-09-11 16:01:53.567202887 +0000 UTC m=+407.285177990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0bc70645-7134-43fd-9294-da3a3651d464-cert") pod "kserve-controller-manager-d6f4b65b8-hwz4v" (UID: "0bc70645-7134-43fd-9294-da3a3651d464") : secret "kserve-webhook-server-cert" not found
Sep 11 16:01:53.075840 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:53.075816    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8l9l\" (UniqueName: \"kubernetes.io/projected/0bc70645-7134-43fd-9294-da3a3651d464-kube-api-access-d8l9l\") pod \"kserve-controller-manager-d6f4b65b8-hwz4v\" (UID: \"0bc70645-7134-43fd-9294-da3a3651d464\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:53.258781 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:53.258752    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:53.570957 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:53.570932    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bc70645-7134-43fd-9294-da3a3651d464-cert\") pod \"kserve-controller-manager-d6f4b65b8-hwz4v\" (UID: \"0bc70645-7134-43fd-9294-da3a3651d464\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:53.573120 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:53.573099    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0bc70645-7134-43fd-9294-da3a3651d464-cert\") pod \"kserve-controller-manager-d6f4b65b8-hwz4v\" (UID: \"0bc70645-7134-43fd-9294-da3a3651d464\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:53.771540 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:53.771515    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:54.054420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:54.054292    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"]
Sep 11 16:01:54.057259 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:01:54.057212    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc70645_7134_43fd_9294_da3a3651d464.slice/crio-a4ed9f8dbcd87e3aa7668e7f29926dfa7690bf8af5f3df13d58594bd7890a7d3 WatchSource:0}: Error finding container a4ed9f8dbcd87e3aa7668e7f29926dfa7690bf8af5f3df13d58594bd7890a7d3: Status 404 returned error can't find the container with id a4ed9f8dbcd87e3aa7668e7f29926dfa7690bf8af5f3df13d58594bd7890a7d3
Sep 11 16:01:54.128653 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:54.128597    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:54.284437 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:54.284409    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v" event={"ID":"0bc70645-7134-43fd-9294-da3a3651d464","Type":"ContainerStarted","Data":"a4ed9f8dbcd87e3aa7668e7f29926dfa7690bf8af5f3df13d58594bd7890a7d3"}
Sep 11 16:01:54.317416 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:54.317387    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:01:57.302976 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:57.302940    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v" event={"ID":"0bc70645-7134-43fd-9294-da3a3651d464","Type":"ContainerStarted","Data":"131aaaa33db2be3ea9b565313264229971a9276a2e33c0e01eef34be9cc321c8"}
Sep 11 16:01:57.303368 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:57.303061    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:01:57.326374 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:57.326322    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v" podStartSLOduration=3.057999278 podStartE2EDuration="5.32630847s" totalImagesPullingTime="2.268309192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:01:57.324402904 +0000 UTC m=+411.042378044" watchObservedRunningTime="2026-09-11 16:01:57.32630847 +0000 UTC m=+411.044283589"
Sep 11 16:01:58.258607 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:01:58.258576    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wgdnm"
Sep 11 16:02:28.311771 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:02:28.311737    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-hwz4v"
Sep 11 16:02:39.984149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:02:39.984030    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:02:40.084447 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:02:40.084411    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:02:43.916479 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:02:43.916444    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:02:44.007626 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:02:44.007537    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:02:52.994134 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:02:52.994096    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:02:54.982309 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:02:54.982272    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:02:55.043889 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:02:55.043855    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:06.106444 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:06.106403    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:06.507054 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:06.506963    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:07.314067 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:07.314036    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:11.594374 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.594341    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"]
Sep 11 16:03:11.596544 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.596526    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:11.599581 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.599551    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xcpqd\""
Sep 11 16:03:11.599703 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.599614    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 16:03:11.599813 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.599795    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 16:03:11.609056 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.609032    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxjkp\" (UniqueName: \"kubernetes.io/projected/d24facb1-2561-4a63-9b7d-f7bccb511f3a-kube-api-access-hxjkp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lrh7h\" (UID: \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:11.609149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.609067    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d24facb1-2561-4a63-9b7d-f7bccb511f3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lrh7h\" (UID: \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:11.626492 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.626468    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"]
Sep 11 16:03:11.709840 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.709807    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxjkp\" (UniqueName: \"kubernetes.io/projected/d24facb1-2561-4a63-9b7d-f7bccb511f3a-kube-api-access-hxjkp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lrh7h\" (UID: \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:11.709940 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.709850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d24facb1-2561-4a63-9b7d-f7bccb511f3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lrh7h\" (UID: \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:11.710136 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.710122    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d24facb1-2561-4a63-9b7d-f7bccb511f3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lrh7h\" (UID: \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:11.720581 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.720555    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxjkp\" (UniqueName: \"kubernetes.io/projected/d24facb1-2561-4a63-9b7d-f7bccb511f3a-kube-api-access-hxjkp\") pod \"kuadrant-operator-controller-manager-76f8f686bf-lrh7h\" (UID: \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:11.857521 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.857449    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:11.906366 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.906342    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:11.981997 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:11.981962    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:12.206635 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:12.206556    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"]
Sep 11 16:03:12.209643 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:03:12.209615    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24facb1_2561_4a63_9b7d_f7bccb511f3a.slice/crio-78e7dd02aaa2f79331888aa9cee14e9ec7d79d9593d34feced87d1a047f12def WatchSource:0}: Error finding container 78e7dd02aaa2f79331888aa9cee14e9ec7d79d9593d34feced87d1a047f12def: Status 404 returned error can't find the container with id 78e7dd02aaa2f79331888aa9cee14e9ec7d79d9593d34feced87d1a047f12def
Sep 11 16:03:12.554527 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:12.554459    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:12.585115 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:12.585085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h" event={"ID":"d24facb1-2561-4a63-9b7d-f7bccb511f3a","Type":"ContainerStarted","Data":"78e7dd02aaa2f79331888aa9cee14e9ec7d79d9593d34feced87d1a047f12def"}
Sep 11 16:03:15.540966 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:15.540925    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:16.606370 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:16.606335    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h" event={"ID":"d24facb1-2561-4a63-9b7d-f7bccb511f3a","Type":"ContainerStarted","Data":"00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56"}
Sep 11 16:03:16.606813 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:16.606474    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:16.654586 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:16.654533    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h" podStartSLOduration=2.138969131 podStartE2EDuration="5.654513413s" totalImagesPullingTime="3.515544282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:16.65174208 +0000 UTC m=+490.369717224" watchObservedRunningTime="2026-09-11 16:03:16.654513413 +0000 UTC m=+490.372488534"
Sep 11 16:03:17.584829 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.584796    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"]
Sep 11 16:03:17.586951 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.586935    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"
Sep 11 16:03:17.589652 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.589632    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rwkr7\""
Sep 11 16:03:17.589970 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.589955    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 16:03:17.599926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.599903    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"]
Sep 11 16:03:17.656127 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.656103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rp86\" (UniqueName: \"kubernetes.io/projected/a5c46727-f6d3-4480-ac58-258c42f80250-kube-api-access-9rp86\") pod \"dns-operator-controller-manager-7b84cd7c9c-ktnfx\" (UID: \"a5c46727-f6d3-4480-ac58-258c42f80250\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"
Sep 11 16:03:17.757191 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.757163    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rp86\" (UniqueName: \"kubernetes.io/projected/a5c46727-f6d3-4480-ac58-258c42f80250-kube-api-access-9rp86\") pod \"dns-operator-controller-manager-7b84cd7c9c-ktnfx\" (UID: \"a5c46727-f6d3-4480-ac58-258c42f80250\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"
Sep 11 16:03:17.767657 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.767635    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rp86\" (UniqueName: \"kubernetes.io/projected/a5c46727-f6d3-4480-ac58-258c42f80250-kube-api-access-9rp86\") pod \"dns-operator-controller-manager-7b84cd7c9c-ktnfx\" (UID: \"a5c46727-f6d3-4480-ac58-258c42f80250\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"
Sep 11 16:03:17.897207 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.897143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"
Sep 11 16:03:17.931310 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:17.931279    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:18.175839 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:18.175816    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"]
Sep 11 16:03:18.178302 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:03:18.178276    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c46727_f6d3_4480_ac58_258c42f80250.slice/crio-1b760e9a40f51de0a98aa9fba159ade6349b55e2e9298a2f10d4af91023953c1 WatchSource:0}: Error finding container 1b760e9a40f51de0a98aa9fba159ade6349b55e2e9298a2f10d4af91023953c1: Status 404 returned error can't find the container with id 1b760e9a40f51de0a98aa9fba159ade6349b55e2e9298a2f10d4af91023953c1
Sep 11 16:03:18.367091 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:18.367047    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:18.505078 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:18.504992    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:18.605619 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:18.605588    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:18.614909 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:18.614884    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx" event={"ID":"a5c46727-f6d3-4480-ac58-258c42f80250","Type":"ContainerStarted","Data":"1b760e9a40f51de0a98aa9fba159ade6349b55e2e9298a2f10d4af91023953c1"}
Sep 11 16:03:19.046480 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:19.046410    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:19.148447 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:19.148397    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:20.626231 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:20.626197    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx" event={"ID":"a5c46727-f6d3-4480-ac58-258c42f80250","Type":"ContainerStarted","Data":"c82f24f98dd4d076ed6b2fb9a9ce0afcb04ddc8120fe654d151691ceaaa12b18"}
Sep 11 16:03:20.626638 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:20.626300    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"
Sep 11 16:03:20.653450 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:20.653405    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx" podStartSLOduration=2.140463598 podStartE2EDuration="3.653392523s" totalImagesPullingTime="1.512928925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:20.652387679 +0000 UTC m=+494.370362837" watchObservedRunningTime="2026-09-11 16:03:20.653392523 +0000 UTC m=+494.371367642"
Sep 11 16:03:21.307156 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:21.307119    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-hpf42"]
Sep 11 16:03:21.309922 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:21.309900    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-hpf42"
Sep 11 16:03:21.312617 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:21.312595    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zqb94\""
Sep 11 16:03:21.320480 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:21.320454    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-hpf42"]
Sep 11 16:03:21.381925 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:21.381899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jrx2\" (UniqueName: \"kubernetes.io/projected/ad68438c-75e2-4db3-ae49-316911d70d17-kube-api-access-6jrx2\") pod \"authorino-operator-76b478bd78-hpf42\" (UID: \"ad68438c-75e2-4db3-ae49-316911d70d17\") " pod="kuadrant-system/authorino-operator-76b478bd78-hpf42"
Sep 11 16:03:21.482758 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:21.482731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jrx2\" (UniqueName: \"kubernetes.io/projected/ad68438c-75e2-4db3-ae49-316911d70d17-kube-api-access-6jrx2\") pod \"authorino-operator-76b478bd78-hpf42\" (UID: \"ad68438c-75e2-4db3-ae49-316911d70d17\") " pod="kuadrant-system/authorino-operator-76b478bd78-hpf42"
Sep 11 16:03:21.491881 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:21.491850    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jrx2\" (UniqueName: \"kubernetes.io/projected/ad68438c-75e2-4db3-ae49-316911d70d17-kube-api-access-6jrx2\") pod \"authorino-operator-76b478bd78-hpf42\" (UID: \"ad68438c-75e2-4db3-ae49-316911d70d17\") " pod="kuadrant-system/authorino-operator-76b478bd78-hpf42"
Sep 11 16:03:21.619790 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:21.619756    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-hpf42"
Sep 11 16:03:21.893796 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:21.893772    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-hpf42"]
Sep 11 16:03:21.896321 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:03:21.896290    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad68438c_75e2_4db3_ae49_316911d70d17.slice/crio-dfd5b28367a22f00145b66b09f7e8623037f70af80aebc39799f578e86146409 WatchSource:0}: Error finding container dfd5b28367a22f00145b66b09f7e8623037f70af80aebc39799f578e86146409: Status 404 returned error can't find the container with id dfd5b28367a22f00145b66b09f7e8623037f70af80aebc39799f578e86146409
Sep 11 16:03:22.638149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:22.638111    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-hpf42" event={"ID":"ad68438c-75e2-4db3-ae49-316911d70d17","Type":"ContainerStarted","Data":"dfd5b28367a22f00145b66b09f7e8623037f70af80aebc39799f578e86146409"}
Sep 11 16:03:23.644320 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:23.644291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-hpf42" event={"ID":"ad68438c-75e2-4db3-ae49-316911d70d17","Type":"ContainerStarted","Data":"ee3a7e3ab89edf3c3e50a8d92e20c98465f8103b8beec17525ba5092ef7effb8"}
Sep 11 16:03:23.644662 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:23.644359    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-hpf42"
Sep 11 16:03:23.662985 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:23.662930    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-hpf42" podStartSLOduration=1.2633501630000001 podStartE2EDuration="2.662913959s" totalImagesPullingTime="1.399563796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:23.662084869 +0000 UTC m=+497.380060008" watchObservedRunningTime="2026-09-11 16:03:23.662913959 +0000 UTC m=+497.380889081"
Sep 11 16:03:24.934578 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:24.934547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:26.217997 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:26.217965    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:26.318453 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:26.318422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:27.613141 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:27.613110    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:30.405306 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.405262    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"]
Sep 11 16:03:30.405751 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.405523    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h" podUID="d24facb1-2561-4a63-9b7d-f7bccb511f3a" containerName="manager" containerID="cri-o://00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56" gracePeriod=2
Sep 11 16:03:30.413093 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.412999    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"]
Sep 11 16:03:30.434484 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.434457    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"]
Sep 11 16:03:30.434750 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.434736    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d24facb1-2561-4a63-9b7d-f7bccb511f3a" containerName="manager"
Sep 11 16:03:30.434750 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.434748    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d24facb1-2561-4a63-9b7d-f7bccb511f3a" containerName="manager"
Sep 11 16:03:30.434879 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.434798    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d24facb1-2561-4a63-9b7d-f7bccb511f3a" containerName="manager"
Sep 11 16:03:30.436969 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.436950    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:30.439415 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:03:30.439387    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lrh7h\" is forbidden: User \"system:node:ip-10-0-141-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-101.ec2.internal' and this object" podUID="d24facb1-2561-4a63-9b7d-f7bccb511f3a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:30.473436 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.473412    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"]
Sep 11 16:03:30.547369 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.547181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020f810-00ba-4dda-b0df-ee170182f268-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qqjhf\" (UID: \"2020f810-00ba-4dda-b0df-ee170182f268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:30.547369 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.547259    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqx2l\" (UniqueName: \"kubernetes.io/projected/2020f810-00ba-4dda-b0df-ee170182f268-kube-api-access-lqx2l\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qqjhf\" (UID: \"2020f810-00ba-4dda-b0df-ee170182f268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:30.634716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.634697    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:30.637893 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:03:30.637870    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lrh7h\" is forbidden: User \"system:node:ip-10-0-141-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-101.ec2.internal' and this object" podUID="d24facb1-2561-4a63-9b7d-f7bccb511f3a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:30.648270 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.648230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqx2l\" (UniqueName: \"kubernetes.io/projected/2020f810-00ba-4dda-b0df-ee170182f268-kube-api-access-lqx2l\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qqjhf\" (UID: \"2020f810-00ba-4dda-b0df-ee170182f268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:30.648372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.648318    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020f810-00ba-4dda-b0df-ee170182f268-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qqjhf\" (UID: \"2020f810-00ba-4dda-b0df-ee170182f268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:30.648587 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.648570    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020f810-00ba-4dda-b0df-ee170182f268-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qqjhf\" (UID: \"2020f810-00ba-4dda-b0df-ee170182f268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:30.677964 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.677913    2564 generic.go:356] "Generic (PLEG): container finished" podID="d24facb1-2561-4a63-9b7d-f7bccb511f3a" containerID="00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56" exitCode=0
Sep 11 16:03:30.677964 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.677950    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:30.678116 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.678023    2564 scope.go:117] "RemoveContainer" containerID="00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56"
Sep 11 16:03:30.680861 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.680836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqx2l\" (UniqueName: \"kubernetes.io/projected/2020f810-00ba-4dda-b0df-ee170182f268-kube-api-access-lqx2l\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qqjhf\" (UID: \"2020f810-00ba-4dda-b0df-ee170182f268\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:30.682066 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:03:30.682036    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-lrh7h\" is forbidden: User \"system:node:ip-10-0-141-101.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-101.ec2.internal' and this object" podUID="d24facb1-2561-4a63-9b7d-f7bccb511f3a" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-lrh7h"
Sep 11 16:03:30.689201 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.689184    2564 scope.go:117] "RemoveContainer" containerID="00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56"
Sep 11 16:03:30.689468 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:03:30.689447    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56\": container with ID starting with 00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56 not found: ID does not exist" containerID="00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56"
Sep 11 16:03:30.689532 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.689479    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56"} err="failed to get container status \"00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56\": rpc error: code = NotFound desc = could not find container \"00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56\": container with ID starting with 00574f401bb0818c447841a863fbb87a36154aa2d753dfe436b0efda328f4f56 not found: ID does not exist"
Sep 11 16:03:30.749249 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.749226    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d24facb1-2561-4a63-9b7d-f7bccb511f3a-extensions-socket-volume\") pod \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\" (UID: \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\") "
Sep 11 16:03:30.749344 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.749297    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxjkp\" (UniqueName: \"kubernetes.io/projected/d24facb1-2561-4a63-9b7d-f7bccb511f3a-kube-api-access-hxjkp\") pod \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\" (UID: \"d24facb1-2561-4a63-9b7d-f7bccb511f3a\") "
Sep 11 16:03:30.749635 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.749613    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24facb1-2561-4a63-9b7d-f7bccb511f3a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d24facb1-2561-4a63-9b7d-f7bccb511f3a" (UID: "d24facb1-2561-4a63-9b7d-f7bccb511f3a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:03:30.751125 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.751105    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24facb1-2561-4a63-9b7d-f7bccb511f3a-kube-api-access-hxjkp" (OuterVolumeSpecName: "kube-api-access-hxjkp") pod "d24facb1-2561-4a63-9b7d-f7bccb511f3a" (UID: "d24facb1-2561-4a63-9b7d-f7bccb511f3a"). InnerVolumeSpecName "kube-api-access-hxjkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:03:30.779293 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.779270    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:30.810660 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.810634    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d24facb1-2561-4a63-9b7d-f7bccb511f3a" path="/var/lib/kubelet/pods/d24facb1-2561-4a63-9b7d-f7bccb511f3a/volumes"
Sep 11 16:03:30.850721 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.850698    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d24facb1-2561-4a63-9b7d-f7bccb511f3a-extensions-socket-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:03:30.850721 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:30.850721    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxjkp\" (UniqueName: \"kubernetes.io/projected/d24facb1-2561-4a63-9b7d-f7bccb511f3a-kube-api-access-hxjkp\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:03:31.054210 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:31.054188    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"]
Sep 11 16:03:31.055592 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:03:31.055573    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2020f810_00ba_4dda_b0df_ee170182f268.slice/crio-0cfe3db7c7e8ff1aaeff703e455dbcc6caf320cbe133a1b065c27c985208404a WatchSource:0}: Error finding container 0cfe3db7c7e8ff1aaeff703e455dbcc6caf320cbe133a1b065c27c985208404a: Status 404 returned error can't find the container with id 0cfe3db7c7e8ff1aaeff703e455dbcc6caf320cbe133a1b065c27c985208404a
Sep 11 16:03:31.634097 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:31.634068    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-ktnfx"
Sep 11 16:03:31.685497 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:31.685455    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf" event={"ID":"2020f810-00ba-4dda-b0df-ee170182f268","Type":"ContainerStarted","Data":"9b17782597a9b897e20fdf3b3b9f59b0a3bd229f5864296c556d33cdd1c8cc62"}
Sep 11 16:03:31.685677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:31.685558    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:31.685677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:31.685583    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf" event={"ID":"2020f810-00ba-4dda-b0df-ee170182f268","Type":"ContainerStarted","Data":"0cfe3db7c7e8ff1aaeff703e455dbcc6caf320cbe133a1b065c27c985208404a"}
Sep 11 16:03:31.706489 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:31.706443    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf" podStartSLOduration=1.7064298839999998 podStartE2EDuration="1.706429884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:31.704953654 +0000 UTC m=+505.422928775" watchObservedRunningTime="2026-09-11 16:03:31.706429884 +0000 UTC m=+505.424405002"
Sep 11 16:03:31.999835 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:31.999761    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:32.120487 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:32.120458    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:33.490146 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:33.490107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:34.650766 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:34.650729    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-hpf42"
Sep 11 16:03:34.661578 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:34.661551    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:35.770457 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:35.770423    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:35.864562 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:35.864530    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:41.721158 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:41.721119    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:41.785644 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:41.785602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:41.928305 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:41.928270    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:41.994318 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:41.994224    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:42.692343 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:42.692285    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:45.508074 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.508037    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"]
Sep 11 16:03:45.508539 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.508264    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf" podUID="2020f810-00ba-4dda-b0df-ee170182f268" containerName="manager" containerID="cri-o://9b17782597a9b897e20fdf3b3b9f59b0a3bd229f5864296c556d33cdd1c8cc62" gracePeriod=10
Sep 11 16:03:45.745195 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.745171    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"]
Sep 11 16:03:45.757970 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.757945    2564 generic.go:356] "Generic (PLEG): container finished" podID="2020f810-00ba-4dda-b0df-ee170182f268" containerID="9b17782597a9b897e20fdf3b3b9f59b0a3bd229f5864296c556d33cdd1c8cc62" exitCode=0
Sep 11 16:03:45.768528 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.768465    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf" event={"ID":"2020f810-00ba-4dda-b0df-ee170182f268","Type":"ContainerDied","Data":"9b17782597a9b897e20fdf3b3b9f59b0a3bd229f5864296c556d33cdd1c8cc62"}
Sep 11 16:03:45.768528 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.768501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf" event={"ID":"2020f810-00ba-4dda-b0df-ee170182f268","Type":"ContainerDied","Data":"0cfe3db7c7e8ff1aaeff703e455dbcc6caf320cbe133a1b065c27c985208404a"}
Sep 11 16:03:45.768528 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.768518    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cfe3db7c7e8ff1aaeff703e455dbcc6caf320cbe133a1b065c27c985208404a"
Sep 11 16:03:45.768528 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.768529    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"]
Sep 11 16:03:45.768743 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.768583    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:03:45.778967 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.778925    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:45.858411 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.858392    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020f810-00ba-4dda-b0df-ee170182f268-extensions-socket-volume\") pod \"2020f810-00ba-4dda-b0df-ee170182f268\" (UID: \"2020f810-00ba-4dda-b0df-ee170182f268\") "
Sep 11 16:03:45.858520 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.858423    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqx2l\" (UniqueName: \"kubernetes.io/projected/2020f810-00ba-4dda-b0df-ee170182f268-kube-api-access-lqx2l\") pod \"2020f810-00ba-4dda-b0df-ee170182f268\" (UID: \"2020f810-00ba-4dda-b0df-ee170182f268\") "
Sep 11 16:03:45.858623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.858605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d5d595f-0fda-4617-bc08-3cf3c4676fac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-htwmw\" (UID: \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:03:45.858683 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.858652    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2rp9\" (UniqueName: \"kubernetes.io/projected/8d5d595f-0fda-4617-bc08-3cf3c4676fac-kube-api-access-s2rp9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-htwmw\" (UID: \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:03:45.858798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.858776    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2020f810-00ba-4dda-b0df-ee170182f268-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2020f810-00ba-4dda-b0df-ee170182f268" (UID: "2020f810-00ba-4dda-b0df-ee170182f268"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:03:45.860572 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.860550    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2020f810-00ba-4dda-b0df-ee170182f268-kube-api-access-lqx2l" (OuterVolumeSpecName: "kube-api-access-lqx2l") pod "2020f810-00ba-4dda-b0df-ee170182f268" (UID: "2020f810-00ba-4dda-b0df-ee170182f268"). InnerVolumeSpecName "kube-api-access-lqx2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:03:45.959534 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.959509    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d5d595f-0fda-4617-bc08-3cf3c4676fac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-htwmw\" (UID: \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:03:45.959623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.959548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2rp9\" (UniqueName: \"kubernetes.io/projected/8d5d595f-0fda-4617-bc08-3cf3c4676fac-kube-api-access-s2rp9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-htwmw\" (UID: \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:03:45.959623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.959589    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2020f810-00ba-4dda-b0df-ee170182f268-extensions-socket-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:03:45.959623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.959598    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqx2l\" (UniqueName: \"kubernetes.io/projected/2020f810-00ba-4dda-b0df-ee170182f268-kube-api-access-lqx2l\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:03:45.959841 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.959823    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d5d595f-0fda-4617-bc08-3cf3c4676fac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-htwmw\" (UID: \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:03:45.968022 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:45.968003    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2rp9\" (UniqueName: \"kubernetes.io/projected/8d5d595f-0fda-4617-bc08-3cf3c4676fac-kube-api-access-s2rp9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-htwmw\" (UID: \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:03:46.077861 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.077820    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:03:46.351663 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.351594    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"]
Sep 11 16:03:46.355138 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:03:46.355106    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5d595f_0fda_4617_bc08_3cf3c4676fac.slice/crio-034dcf959daf999a3cae328752100df153713c1ed2b7dbd1c261f70d0ffa022e WatchSource:0}: Error finding container 034dcf959daf999a3cae328752100df153713c1ed2b7dbd1c261f70d0ffa022e: Status 404 returned error can't find the container with id 034dcf959daf999a3cae328752100df153713c1ed2b7dbd1c261f70d0ffa022e
Sep 11 16:03:46.764158 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.764086    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw" event={"ID":"8d5d595f-0fda-4617-bc08-3cf3c4676fac","Type":"ContainerStarted","Data":"8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5"}
Sep 11 16:03:46.764158 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.764141    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw" event={"ID":"8d5d595f-0fda-4617-bc08-3cf3c4676fac","Type":"ContainerStarted","Data":"034dcf959daf999a3cae328752100df153713c1ed2b7dbd1c261f70d0ffa022e"}
Sep 11 16:03:46.764570 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.764264    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:03:46.764570 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.764307    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"
Sep 11 16:03:46.768553 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.768527    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:46.799996 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.799958    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw" podStartSLOduration=1.799946041 podStartE2EDuration="1.799946041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:03:46.790463155 +0000 UTC m=+520.508438274" watchObservedRunningTime="2026-09-11 16:03:46.799946041 +0000 UTC m=+520.517921162"
Sep 11 16:03:46.816909 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.816885    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"]
Sep 11 16:03:46.830605 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.830578    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qqjhf"]
Sep 11 16:03:46.926758 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:46.926731    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:47.155132 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:47.155104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:48.815374 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:48.815334    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2020f810-00ba-4dda-b0df-ee170182f268" path="/var/lib/kubelet/pods/2020f810-00ba-4dda-b0df-ee170182f268/volumes"
Sep 11 16:03:54.902120 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:54.902075    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:55.025835 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:55.025805    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:56.313523 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:56.313485    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:56.412926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:56.412896    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:03:57.772109 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:03:57.772077    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:04:02.274024 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:02.273947    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:02.798652 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:02.798615    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:02.917540 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:02.917506    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:03.036741 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:03.036709    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:08.023363 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:08.023320    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:08.097816 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:08.097783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:08.414691 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:08.414659    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:08.639565 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:08.639537    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:08.742258 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:08.742166    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:10.093205 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:10.093159    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:10.194002 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:10.193969    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:15.608188 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:15.608148    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:15.708548 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:15.708512    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:17.349884 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:17.349847    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:17.487218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:17.487189    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:17.587860 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:17.587824    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:18.136173 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:18.136138    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:18.236183 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:18.236158    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:18.331947 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:18.331923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:24.640857 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:24.640818    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:24.755452 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:24.755413    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:27.297644 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:27.297602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:36.549489 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:36.549455    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:36.653643 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:36.653609    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:37.124650 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:37.124617    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:48.674216 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:48.674178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:48.741928 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:48.741897    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:56.776975 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.776937    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58fdfdff-cw2x7"]
Sep 11 16:04:56.777442 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.777377    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2020f810-00ba-4dda-b0df-ee170182f268" containerName="manager"
Sep 11 16:04:56.777442 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.777395    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2020f810-00ba-4dda-b0df-ee170182f268" containerName="manager"
Sep 11 16:04:56.777548 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.777468    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2020f810-00ba-4dda-b0df-ee170182f268" containerName="manager"
Sep 11 16:04:56.779440 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.779421    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:56.782114 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.782093    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 16:04:56.782259 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.782227    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w4bfh\""
Sep 11 16:04:56.782334 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.782325    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 16:04:56.783388 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.783372    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 16:04:56.791286 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.791266    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58fdfdff-cw2x7"]
Sep 11 16:04:56.860773 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.860754    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-metrics-certs\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:56.860902 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.860792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-webhook-cert\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:56.860902 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.860876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q5nd\" (UniqueName: \"kubernetes.io/projected/b426a586-698d-4dfb-a648-a656a87cb5aa-kube-api-access-5q5nd\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:56.962024 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.961987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-metrics-certs\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:56.962144 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.962035    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-webhook-cert\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:56.962144 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.962100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q5nd\" (UniqueName: \"kubernetes.io/projected/b426a586-698d-4dfb-a648-a656a87cb5aa-kube-api-access-5q5nd\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:56.962280 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:04:56.962160    2564 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 11 16:04:56.962280 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:04:56.962226    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-webhook-cert podName:b426a586-698d-4dfb-a648-a656a87cb5aa nodeName:}" failed. No retries permitted until 2026-09-11 16:04:57.462210489 +0000 UTC m=+591.180185587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-webhook-cert") pod "maas-controller-58fdfdff-cw2x7" (UID: "b426a586-698d-4dfb-a648-a656a87cb5aa") : secret "maas-controller-webhook-cert" not found
Sep 11 16:04:56.964555 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.964535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-metrics-certs\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:56.975428 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.975409    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q5nd\" (UniqueName: \"kubernetes.io/projected/b426a586-698d-4dfb-a648-a656a87cb5aa-kube-api-access-5q5nd\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:56.977930 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.977911    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f55cd5c49-fz5g4"]
Sep 11 16:04:56.980017 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.979997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:56.990563 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:56.990541    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-fz5g4"]
Sep 11 16:04:57.063231 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.063199    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7hx\" (UniqueName: \"kubernetes.io/projected/b310a818-e4e7-4463-b814-2ac0f9d6450c-kube-api-access-sj7hx\") pod \"maas-controller-f55cd5c49-fz5g4\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") " pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.063346 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.063313    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-webhook-cert\") pod \"maas-controller-f55cd5c49-fz5g4\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") " pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.063404 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.063354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-metrics-certs\") pod \"maas-controller-f55cd5c49-fz5g4\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") " pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.164650 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.164628    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-webhook-cert\") pod \"maas-controller-f55cd5c49-fz5g4\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") " pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.164770 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.164663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-metrics-certs\") pod \"maas-controller-f55cd5c49-fz5g4\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") " pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.164770 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.164694    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7hx\" (UniqueName: \"kubernetes.io/projected/b310a818-e4e7-4463-b814-2ac0f9d6450c-kube-api-access-sj7hx\") pod \"maas-controller-f55cd5c49-fz5g4\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") " pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.167217 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.167192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-metrics-certs\") pod \"maas-controller-f55cd5c49-fz5g4\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") " pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.167316 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.167261    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-webhook-cert\") pod \"maas-controller-f55cd5c49-fz5g4\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") " pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.176845 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.176825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7hx\" (UniqueName: \"kubernetes.io/projected/b310a818-e4e7-4463-b814-2ac0f9d6450c-kube-api-access-sj7hx\") pod \"maas-controller-f55cd5c49-fz5g4\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") " pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.289467 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.289440    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:04:57.467771 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.467740    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-webhook-cert\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:57.469878 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.469860    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-webhook-cert\") pod \"maas-controller-58fdfdff-cw2x7\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") " pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:57.564610 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.564579    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-fz5g4"]
Sep 11 16:04:57.571034 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:04:57.570997    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb310a818_e4e7_4463_b814_2ac0f9d6450c.slice/crio-0b2ddc8c81c38a6e5eed409e1b7baee01940d2f93dadb766af144df4f527dcd0 WatchSource:0}: Error finding container 0b2ddc8c81c38a6e5eed409e1b7baee01940d2f93dadb766af144df4f527dcd0: Status 404 returned error can't find the container with id 0b2ddc8c81c38a6e5eed409e1b7baee01940d2f93dadb766af144df4f527dcd0
Sep 11 16:04:57.689153 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.689089    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:04:57.732402 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.732348    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:57.831576 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.831549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:04:57.960975 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:57.960953    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58fdfdff-cw2x7"]
Sep 11 16:04:57.963078 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:04:57.963057    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb426a586_698d_4dfb_a648_a656a87cb5aa.slice/crio-1203c0ca1eb57d50af1ef468b1b5090f4b989ccb203b388f619298580775af0e WatchSource:0}: Error finding container 1203c0ca1eb57d50af1ef468b1b5090f4b989ccb203b388f619298580775af0e: Status 404 returned error can't find the container with id 1203c0ca1eb57d50af1ef468b1b5090f4b989ccb203b388f619298580775af0e
Sep 11 16:04:58.069497 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:58.069466    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fdfdff-cw2x7" event={"ID":"b426a586-698d-4dfb-a648-a656a87cb5aa","Type":"ContainerStarted","Data":"1203c0ca1eb57d50af1ef468b1b5090f4b989ccb203b388f619298580775af0e"}
Sep 11 16:04:58.070560 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:04:58.070538    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-fz5g4" event={"ID":"b310a818-e4e7-4463-b814-2ac0f9d6450c","Type":"ContainerStarted","Data":"0b2ddc8c81c38a6e5eed409e1b7baee01940d2f93dadb766af144df4f527dcd0"}
Sep 11 16:05:00.031846 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:00.031790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:00.081569 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:00.081484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-fz5g4" event={"ID":"b310a818-e4e7-4463-b814-2ac0f9d6450c","Type":"ContainerStarted","Data":"952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457"}
Sep 11 16:05:00.081718 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:00.081572    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:05:00.082898 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:00.082871    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fdfdff-cw2x7" event={"ID":"b426a586-698d-4dfb-a648-a656a87cb5aa","Type":"ContainerStarted","Data":"392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6"}
Sep 11 16:05:00.083018 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:00.082993    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:05:00.106116 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:00.106071    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f55cd5c49-fz5g4" podStartSLOduration=1.852709274 podStartE2EDuration="4.106056815s" totalImagesPullingTime="2.253347541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:05:00.104709648 +0000 UTC m=+593.822684779" watchObservedRunningTime="2026-09-11 16:05:00.106056815 +0000 UTC m=+593.824031936"
Sep 11 16:05:00.125190 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:00.125151    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58fdfdff-cw2x7" podStartSLOduration=2.205960812 podStartE2EDuration="4.125136496s" totalImagesPullingTime="1.919175684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:05:00.124566367 +0000 UTC m=+593.842541508" watchObservedRunningTime="2026-09-11 16:05:00.125136496 +0000 UTC m=+593.843111616"
Sep 11 16:05:00.129929 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:00.129906    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:08.574911 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:08.574870    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:11.089260 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.089204    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:05:11.089700 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.089375    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:05:11.145877 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.145841    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58fdfdff-cw2x7"]
Sep 11 16:05:11.146086 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.146035    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58fdfdff-cw2x7" podUID="b426a586-698d-4dfb-a648-a656a87cb5aa" containerName="manager" containerID="cri-o://392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6" gracePeriod=30
Sep 11 16:05:11.375566 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.375542    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:05:11.471222 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.471198    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-metrics-certs\") pod \"b426a586-698d-4dfb-a648-a656a87cb5aa\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") "
Sep 11 16:05:11.471369 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.471281    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-webhook-cert\") pod \"b426a586-698d-4dfb-a648-a656a87cb5aa\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") "
Sep 11 16:05:11.471369 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.471322    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q5nd\" (UniqueName: \"kubernetes.io/projected/b426a586-698d-4dfb-a648-a656a87cb5aa-kube-api-access-5q5nd\") pod \"b426a586-698d-4dfb-a648-a656a87cb5aa\" (UID: \"b426a586-698d-4dfb-a648-a656a87cb5aa\") "
Sep 11 16:05:11.473372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.473338    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b426a586-698d-4dfb-a648-a656a87cb5aa" (UID: "b426a586-698d-4dfb-a648-a656a87cb5aa"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:05:11.473372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.473362    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b426a586-698d-4dfb-a648-a656a87cb5aa" (UID: "b426a586-698d-4dfb-a648-a656a87cb5aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:05:11.473505 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.473433    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b426a586-698d-4dfb-a648-a656a87cb5aa-kube-api-access-5q5nd" (OuterVolumeSpecName: "kube-api-access-5q5nd") pod "b426a586-698d-4dfb-a648-a656a87cb5aa" (UID: "b426a586-698d-4dfb-a648-a656a87cb5aa"). InnerVolumeSpecName "kube-api-access-5q5nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:05:11.571910 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.571887    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-webhook-cert\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:05:11.571910 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.571908    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q5nd\" (UniqueName: \"kubernetes.io/projected/b426a586-698d-4dfb-a648-a656a87cb5aa-kube-api-access-5q5nd\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:05:11.572074 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:11.571918    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b426a586-698d-4dfb-a648-a656a87cb5aa-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:05:12.136592 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.136550    2564 generic.go:356] "Generic (PLEG): container finished" podID="b426a586-698d-4dfb-a648-a656a87cb5aa" containerID="392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6" exitCode=0
Sep 11 16:05:12.137030 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.136612    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58fdfdff-cw2x7"
Sep 11 16:05:12.137030 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.136621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fdfdff-cw2x7" event={"ID":"b426a586-698d-4dfb-a648-a656a87cb5aa","Type":"ContainerDied","Data":"392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6"}
Sep 11 16:05:12.137030 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.136670    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58fdfdff-cw2x7" event={"ID":"b426a586-698d-4dfb-a648-a656a87cb5aa","Type":"ContainerDied","Data":"1203c0ca1eb57d50af1ef468b1b5090f4b989ccb203b388f619298580775af0e"}
Sep 11 16:05:12.137030 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.136694    2564 scope.go:117] "RemoveContainer" containerID="392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6"
Sep 11 16:05:12.146442 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.146421    2564 scope.go:117] "RemoveContainer" containerID="392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6"
Sep 11 16:05:12.146764 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:05:12.146746    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6\": container with ID starting with 392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6 not found: ID does not exist" containerID="392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6"
Sep 11 16:05:12.146816 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.146771    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6"} err="failed to get container status \"392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6\": rpc error: code = NotFound desc = could not find container \"392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6\": container with ID starting with 392fc86a904d2275eab75cd37190ba2a69654dc04ceab9a0c9b0e45b2c65b6c6 not found: ID does not exist"
Sep 11 16:05:12.159064 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.159036    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58fdfdff-cw2x7"]
Sep 11 16:05:12.164091 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.164066    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58fdfdff-cw2x7"]
Sep 11 16:05:12.811449 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:12.811417    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b426a586-698d-4dfb-a648-a656a87cb5aa" path="/var/lib/kubelet/pods/b426a586-698d-4dfb-a648-a656a87cb5aa/volumes"
Sep 11 16:05:32.795387 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:32.795293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:33.010750 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:33.010712    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:34.223716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.223672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:34.330698 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.330658    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:34.342991 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.342965    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-s49th"]
Sep 11 16:05:34.343289 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.343276    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b426a586-698d-4dfb-a648-a656a87cb5aa" containerName="manager"
Sep 11 16:05:34.343289 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.343290    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b426a586-698d-4dfb-a648-a656a87cb5aa" containerName="manager"
Sep 11 16:05:34.343420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.343341    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b426a586-698d-4dfb-a648-a656a87cb5aa" containerName="manager"
Sep 11 16:05:34.377129 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.377099    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-s49th"]
Sep 11 16:05:34.377271 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.377204    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.380425 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.380404    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 16:05:34.380894 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.380877    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 16:05:34.381662 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.381647    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 16:05:34.381724 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.381694    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nfbz8\""
Sep 11 16:05:34.381782 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.381655    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 16:05:34.531777 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.531718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-maas-api-tls\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.531777 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.531755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-openshift-service-ca\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.531918 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.531781    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2wnm\" (UniqueName: \"kubernetes.io/projected/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-kube-api-access-k2wnm\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.531918 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.531813    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-metrics-certs\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.609596 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.609568    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:34.633125 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.633101    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-openshift-service-ca\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.633201 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.633141    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2wnm\" (UniqueName: \"kubernetes.io/projected/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-kube-api-access-k2wnm\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.633201 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.633180    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-metrics-certs\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.633305 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.633221    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-maas-api-tls\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.633348 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:05:34.633331    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 11 16:05:34.633419 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:05:34.633406    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-metrics-certs podName:efc49dec-e6b6-44bc-9bf4-bb4b19b24b84 nodeName:}" failed. No retries permitted until 2026-09-11 16:05:35.133386689 +0000 UTC m=+628.851361792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-metrics-certs") pod "maas-api-585675b678-s49th" (UID: "efc49dec-e6b6-44bc-9bf4-bb4b19b24b84") : secret "maas-api-metrics-tls" not found
Sep 11 16:05:34.633811 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.633790    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-openshift-service-ca\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.635556 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.635535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-maas-api-tls\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.645077 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.645054    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2wnm\" (UniqueName: \"kubernetes.io/projected/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-kube-api-access-k2wnm\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:34.836833 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.836800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:34.980119 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:34.980084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:35.081539 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.081512    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:35.137375 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.137313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-metrics-certs\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:35.139708 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.139689    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-metrics-certs\") pod \"maas-api-585675b678-s49th\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") " pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:35.287833 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.287805    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:35.291446 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.291424    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-8slbf"]
Sep 11 16:05:35.293847 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.293831    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:35.297483 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.297455    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 16:05:35.297543 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.297509    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7mtgg\""
Sep 11 16:05:35.313644 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.313614    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-8slbf"]
Sep 11 16:05:35.440682 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.440616    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c21252-78b5-43ab-b332-3922c4d50b79-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-8slbf\" (UID: \"66c21252-78b5-43ab-b332-3922c4d50b79\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:35.440682 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.440670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcd84\" (UniqueName: \"kubernetes.io/projected/66c21252-78b5-43ab-b332-3922c4d50b79-kube-api-access-lcd84\") pod \"payload-processing-5b54b7fbb7-8slbf\" (UID: \"66c21252-78b5-43ab-b332-3922c4d50b79\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:35.541712 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.541680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c21252-78b5-43ab-b332-3922c4d50b79-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-8slbf\" (UID: \"66c21252-78b5-43ab-b332-3922c4d50b79\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:35.541851 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.541726    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcd84\" (UniqueName: \"kubernetes.io/projected/66c21252-78b5-43ab-b332-3922c4d50b79-kube-api-access-lcd84\") pod \"payload-processing-5b54b7fbb7-8slbf\" (UID: \"66c21252-78b5-43ab-b332-3922c4d50b79\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:35.542615 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.542596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66c21252-78b5-43ab-b332-3922c4d50b79-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-8slbf\" (UID: \"66c21252-78b5-43ab-b332-3922c4d50b79\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:35.559717 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.559695    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcd84\" (UniqueName: \"kubernetes.io/projected/66c21252-78b5-43ab-b332-3922c4d50b79-kube-api-access-lcd84\") pod \"payload-processing-5b54b7fbb7-8slbf\" (UID: \"66c21252-78b5-43ab-b332-3922c4d50b79\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:35.573360 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.573341    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-s49th"]
Sep 11 16:05:35.575970 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:05:35.575942    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc49dec_e6b6_44bc_9bf4_bb4b19b24b84.slice/crio-a7df10d53345c607e293a6c89415172ccd07890ada7d05747710123a775b997b WatchSource:0}: Error finding container a7df10d53345c607e293a6c89415172ccd07890ada7d05747710123a775b997b: Status 404 returned error can't find the container with id a7df10d53345c607e293a6c89415172ccd07890ada7d05747710123a775b997b
Sep 11 16:05:35.619152 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.619121    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:35.896186 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:35.896163    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-8slbf"]
Sep 11 16:05:35.898661 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:05:35.898633    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c21252_78b5_43ab_b332_3922c4d50b79.slice/crio-87d47f626a5e641c882e793692fdaa982f66d86803566f1019568a979100964d WatchSource:0}: Error finding container 87d47f626a5e641c882e793692fdaa982f66d86803566f1019568a979100964d: Status 404 returned error can't find the container with id 87d47f626a5e641c882e793692fdaa982f66d86803566f1019568a979100964d
Sep 11 16:05:36.248199 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:36.248106    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf" event={"ID":"66c21252-78b5-43ab-b332-3922c4d50b79","Type":"ContainerStarted","Data":"87d47f626a5e641c882e793692fdaa982f66d86803566f1019568a979100964d"}
Sep 11 16:05:36.249277 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:36.249235    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th" event={"ID":"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84","Type":"ContainerStarted","Data":"a7df10d53345c607e293a6c89415172ccd07890ada7d05747710123a775b997b"}
Sep 11 16:05:38.261717 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:38.261635    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf" event={"ID":"66c21252-78b5-43ab-b332-3922c4d50b79","Type":"ContainerStarted","Data":"05f416aabdd04e6500b9706ec72365d4dd4ee910873c1029090dfbb1a0fd6c7b"}
Sep 11 16:05:38.262203 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:38.261790    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:39.267423 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:39.267342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th" event={"ID":"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84","Type":"ContainerStarted","Data":"ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1"}
Sep 11 16:05:39.267423 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:39.267405    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:39.288731 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:39.287921    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th" podStartSLOduration=1.996113979 podStartE2EDuration="5.287550864s" totalImagesPullingTime="3.291436885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:05:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:05:39.286797145 +0000 UTC m=+633.004772303" watchObservedRunningTime="2026-09-11 16:05:39.287550864 +0000 UTC m=+633.005525986"
Sep 11 16:05:39.288731 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:39.288204    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf" podStartSLOduration=2.658272321 podStartE2EDuration="4.288192502s" totalImagesPullingTime="1.629920181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:05:38.287459289 +0000 UTC m=+632.005434433" watchObservedRunningTime="2026-09-11 16:05:39.288192502 +0000 UTC m=+633.006167623"
Sep 11 16:05:45.277169 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:45.277129    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:05:46.379786 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:46.379747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:46.481987 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:46.481951    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:49.270372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:49.270337    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-8slbf"
Sep 11 16:05:55.519833 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:55.519783    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:05:58.578546 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:05:58.578513    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:06:06.356183 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:06.356136    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:06:09.183952 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.183914    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"]
Sep 11 16:06:09.186978 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.186952    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.189694 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.189657    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-sxmds\""
Sep 11 16:06:09.190788 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.190763    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 16:06:09.190788 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.190780    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 16:06:09.190944 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.190867    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 16:06:09.204973 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.204940    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"]
Sep 11 16:06:09.234455 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.234430    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:06:09.288199 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.288178    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.288309 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.288208    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.288309 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.288262    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.288309 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.288297    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.288416 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.288319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.288416 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.288366    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvqqh\" (UniqueName: \"kubernetes.io/projected/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-kube-api-access-pvqqh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.389531 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.389506    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvqqh\" (UniqueName: \"kubernetes.io/projected/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-kube-api-access-pvqqh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.389629 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.389536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.389629 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.389556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.389629 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.389600    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.389766 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.389627    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.389766 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.389666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.389990 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.389967    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.390067 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.389988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.390123 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.390083    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.391863 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.391842    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.391939 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.391917    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.398872 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.398849    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvqqh\" (UniqueName: \"kubernetes.io/projected/c5b11a2e-b9f3-4da4-bf9a-abdffffc111c-kube-api-access-pvqqh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-7hflj\" (UID: \"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.399094 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.399072    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:06:09.498873 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.498821    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:09.499983 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.499954    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:06:09.783594 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.783571    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"]
Sep 11 16:06:09.785862 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:06:09.785833    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b11a2e_b9f3_4da4_bf9a_abdffffc111c.slice/crio-19a91301801354ec83beeb755e72fe5863ed90163f48172e17288b0153c8f150 WatchSource:0}: Error finding container 19a91301801354ec83beeb755e72fe5863ed90163f48172e17288b0153c8f150: Status 404 returned error can't find the container with id 19a91301801354ec83beeb755e72fe5863ed90163f48172e17288b0153c8f150
Sep 11 16:06:09.787495 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:09.787477    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:06:10.412778 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:10.412739    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj" event={"ID":"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c","Type":"ContainerStarted","Data":"19a91301801354ec83beeb755e72fe5863ed90163f48172e17288b0153c8f150"}
Sep 11 16:06:11.419047 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:11.419012    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj" event={"ID":"c5b11a2e-b9f3-4da4-bf9a-abdffffc111c","Type":"ContainerStarted","Data":"350bb72ea1a44dc45e8a1bf72ff88266d052ebaeb66491e6d1d067822ab0c7fa"}
Sep 11 16:06:11.440049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:11.440006    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj" podStartSLOduration=1.454322716 podStartE2EDuration="2.439992224s" totalImagesPullingTime="985.669508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:06:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:06:11.439081689 +0000 UTC m=+665.157056827" watchObservedRunningTime="2026-09-11 16:06:11.439992224 +0000 UTC m=+665.157967344"
Sep 11 16:06:19.499782 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:19.499748    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:19.500306 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:19.499815    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:19.512332 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:19.512306    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:20.469559 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:20.469526    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-7hflj"
Sep 11 16:06:29.129082 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:29.129047    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:06:47.033875 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.033845    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"]
Sep 11 16:06:47.036466 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.036448    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.039852 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.039834    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 16:06:47.061786 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.061759    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"]
Sep 11 16:06:47.115020 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.114996    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:06:47.161442 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.161414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.161542 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.161454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.161542 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.161477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.161615 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.161536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wln\" (UniqueName: \"kubernetes.io/projected/5780847c-b0e1-4308-b49f-71980aad43e4-kube-api-access-95wln\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.161615 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.161606    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5780847c-b0e1-4308-b49f-71980aad43e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.161683 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.161625    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.262267 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.262224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5780847c-b0e1-4308-b49f-71980aad43e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.262366 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.262316    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.262426 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.262396    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.262426 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.262422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.262533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.262442    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.262533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.262495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wln\" (UniqueName: \"kubernetes.io/projected/5780847c-b0e1-4308-b49f-71980aad43e4-kube-api-access-95wln\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.262767 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.262740    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.262856 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.262778    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.262898 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.262862    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.264524 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.264495    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5780847c-b0e1-4308-b49f-71980aad43e4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.264636 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.264620    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5780847c-b0e1-4308-b49f-71980aad43e4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.275926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.275905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wln\" (UniqueName: \"kubernetes.io/projected/5780847c-b0e1-4308-b49f-71980aad43e4-kube-api-access-95wln\") pod \"e2e-trlp-test-simulated-kserve-598466548d-jdflt\" (UID: \"5780847c-b0e1-4308-b49f-71980aad43e4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.345926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.345905    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:47.618435 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:47.618373    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"]
Sep 11 16:06:47.621406 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:06:47.621374    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5780847c_b0e1_4308_b49f_71980aad43e4.slice/crio-b32f4e315422798d107731dc2fa6eae08582669e068d7b8bc35ab37c2b8bfce3 WatchSource:0}: Error finding container b32f4e315422798d107731dc2fa6eae08582669e068d7b8bc35ab37c2b8bfce3: Status 404 returned error can't find the container with id b32f4e315422798d107731dc2fa6eae08582669e068d7b8bc35ab37c2b8bfce3
Sep 11 16:06:48.592255 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:48.592216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt" event={"ID":"5780847c-b0e1-4308-b49f-71980aad43e4","Type":"ContainerStarted","Data":"dc61bc13a4a2e20adbacb6e9b8ca894b4f347ff74640a2412053c44d900a42e6"}
Sep 11 16:06:48.592711 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:48.592325    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt" event={"ID":"5780847c-b0e1-4308-b49f-71980aad43e4","Type":"ContainerStarted","Data":"b32f4e315422798d107731dc2fa6eae08582669e068d7b8bc35ab37c2b8bfce3"}
Sep 11 16:06:57.346964 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:57.346920    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:57.347435 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:57.346992    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:57.359596 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:57.359560    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:06:57.381004 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:57.380956    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt" podStartSLOduration=11.207434516 podStartE2EDuration="11.380938314s" totalImagesPullingTime="173.503798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:06:48.63281378 +0000 UTC m=+702.350788899" watchObservedRunningTime="2026-09-11 16:06:57.380938314 +0000 UTC m=+711.098913435"
Sep 11 16:06:57.644746 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:06:57.644669    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-jdflt"
Sep 11 16:07:04.645118 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:04.645024    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:07:09.307149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.307111    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"]
Sep 11 16:07:09.309866 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.309847    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.312707 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.312685    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 16:07:09.321323 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.321301    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"]
Sep 11 16:07:09.346976 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.346953    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:07:09.417671 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.417649    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.417671 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.417677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.417859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.417700    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9ks\" (UniqueName: \"kubernetes.io/projected/8e2426ef-aac7-447e-8d93-a37f46ba585d-kube-api-access-wn9ks\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.417859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.417719    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.417859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.417785    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2426ef-aac7-447e-8d93-a37f46ba585d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.417859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.417847    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.518457 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.518435    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.518582 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.518473    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.518582 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.518491    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.518582 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.518514    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9ks\" (UniqueName: \"kubernetes.io/projected/8e2426ef-aac7-447e-8d93-a37f46ba585d-kube-api-access-wn9ks\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.518582 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.518539    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.518582 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.518567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2426ef-aac7-447e-8d93-a37f46ba585d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.518920 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.518898    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.518920 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.518911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.519006 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.518958    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.520567 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.520547    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2426ef-aac7-447e-8d93-a37f46ba585d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.520846 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.520830    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2426ef-aac7-447e-8d93-a37f46ba585d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.533169 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.533151    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9ks\" (UniqueName: \"kubernetes.io/projected/8e2426ef-aac7-447e-8d93-a37f46ba585d-kube-api-access-wn9ks\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf24p\" (UID: \"8e2426ef-aac7-447e-8d93-a37f46ba585d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.620497 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.620476    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:09.893994 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:09.893970    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"]
Sep 11 16:07:09.896166 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:07:09.896139    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2426ef_aac7_447e_8d93_a37f46ba585d.slice/crio-915e7602b2642a54d05fefe061889a5f23e20b4fa9f0b53728d6df72facd4a58 WatchSource:0}: Error finding container 915e7602b2642a54d05fefe061889a5f23e20b4fa9f0b53728d6df72facd4a58: Status 404 returned error can't find the container with id 915e7602b2642a54d05fefe061889a5f23e20b4fa9f0b53728d6df72facd4a58
Sep 11 16:07:10.693949 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:10.693896    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p" event={"ID":"8e2426ef-aac7-447e-8d93-a37f46ba585d","Type":"ContainerStarted","Data":"7a323390fdc76d1fcd63e086fde823e18916e551d8740435fc548cac31a90168"}
Sep 11 16:07:10.694422 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:10.693975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p" event={"ID":"8e2426ef-aac7-447e-8d93-a37f46ba585d","Type":"ContainerStarted","Data":"915e7602b2642a54d05fefe061889a5f23e20b4fa9f0b53728d6df72facd4a58"}
Sep 11 16:07:10.716257 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:10.716197    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p" podStartSLOduration=1.5591669719999999 podStartE2EDuration="1.716183462s" totalImagesPullingTime="157.01649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:07:10.715912138 +0000 UTC m=+724.433887278" watchObservedRunningTime="2026-09-11 16:07:10.716183462 +0000 UTC m=+724.434158580"
Sep 11 16:07:19.621204 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:19.621158    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:19.621660 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:19.621284    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:19.633740 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:19.633709    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:19.745181 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:19.745156    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf24p"
Sep 11 16:07:31.033495 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:07:31.033458    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:16.472951 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:16.472918    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:16.563836 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:16.563809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:17.595711 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:17.595667    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:17.691202 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:17.691175    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:33.421509 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:33.421472    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:33.542544 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:33.542510    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:33.773713 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:33.773615    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:33.862220 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:33.862174    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:34.884380 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:34.884344    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:34.984014 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:34.983988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:35.351692 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:35.351645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:37.071585 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:37.071549    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:37.220294 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:37.220254    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:38.187101 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.187061    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"]
Sep 11 16:09:38.190801 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.190777    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.193653 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.193629    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8e3121-serving-cert\""
Sep 11 16:09:38.204283 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.204229    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"]
Sep 11 16:09:38.213502 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.213482    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-maas-api-tls\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.213587 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.213518    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-metrics-certs\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.213587 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.213567    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhtrd\" (UniqueName: \"kubernetes.io/projected/34bc2cca-1857-4e31-ae3a-659a4af87db7-kube-api-access-vhtrd\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.213660 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.213604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bc2cca-1857-4e31-ae3a-659a4af87db7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.235719 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.235692    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:38.314925 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.314900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-metrics-certs\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.315033 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.314944    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhtrd\" (UniqueName: \"kubernetes.io/projected/34bc2cca-1857-4e31-ae3a-659a4af87db7-kube-api-access-vhtrd\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.315033 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.314966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bc2cca-1857-4e31-ae3a-659a4af87db7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.315033 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.315007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-maas-api-tls\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.315142 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:09:38.315081    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-serving-cert: secret "maas-api-e2e-inf-a-8e3121-serving-cert" not found
Sep 11 16:09:38.315142 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:09:38.315138    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-maas-api-tls podName:34bc2cca-1857-4e31-ae3a-659a4af87db7 nodeName:}" failed. No retries permitted until 2026-09-11 16:09:38.81512376 +0000 UTC m=+872.533098858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-maas-api-tls") pod "maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t" (UID: "34bc2cca-1857-4e31-ae3a-659a4af87db7") : secret "maas-api-e2e-inf-a-8e3121-serving-cert" not found
Sep 11 16:09:38.315626 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.315596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bc2cca-1857-4e31-ae3a-659a4af87db7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.317401 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.317380    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-metrics-certs\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.328774 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.328751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhtrd\" (UniqueName: \"kubernetes.io/projected/34bc2cca-1857-4e31-ae3a-659a4af87db7-kube-api-access-vhtrd\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.342939 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.342913    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:38.631582 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.631555    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:38.816336 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.816274    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:38.819462 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.819439    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-maas-api-tls\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.821705 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.821681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-maas-api-tls\") pod \"maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:38.896831 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:38.896771    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:39.101796 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:39.101764    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:39.374520 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:39.374496    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"]
Sep 11 16:09:39.376557 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:09:39.376525    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34bc2cca_1857_4e31_ae3a_659a4af87db7.slice/crio-02752451b5432feee474cda6475f411f02bd7d2d656a679d35d19f1246782e95 WatchSource:0}: Error finding container 02752451b5432feee474cda6475f411f02bd7d2d656a679d35d19f1246782e95: Status 404 returned error can't find the container with id 02752451b5432feee474cda6475f411f02bd7d2d656a679d35d19f1246782e95
Sep 11 16:09:39.390607 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:39.390575    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t" event={"ID":"34bc2cca-1857-4e31-ae3a-659a4af87db7","Type":"ContainerStarted","Data":"02752451b5432feee474cda6475f411f02bd7d2d656a679d35d19f1246782e95"}
Sep 11 16:09:40.396807 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:40.396774    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t" event={"ID":"34bc2cca-1857-4e31-ae3a-659a4af87db7","Type":"ContainerStarted","Data":"1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6"}
Sep 11 16:09:40.397189 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:40.396837    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:40.418874 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:40.418826    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t" podStartSLOduration=1.84544001 podStartE2EDuration="2.418810303s" totalImagesPullingTime="573.370293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:40.417412236 +0000 UTC m=+874.135387390" watchObservedRunningTime="2026-09-11 16:09:40.418810303 +0000 UTC m=+874.136785422"
Sep 11 16:09:41.403251 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:41.403210    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:41.505550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:41.505523    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:41.637726 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:41.637695    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:42.013303 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:42.013264    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:42.115021 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:42.114989    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:42.250224 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:42.250193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:42.609967 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:42.609917    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:42.701303 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:42.701272    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:43.768392 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:43.768357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:44.485584 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:44.485547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:44.577974 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:44.577945    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:44.679775 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:44.679744    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:45.650502 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:45.650470    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:46.198895 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:46.198863    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:46.299457 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:46.299417    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:46.407488 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:46.407462    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:09:46.946015 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:46.945982    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:47.807492 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.807459    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"]
Sep 11 16:09:47.811537 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.811424    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.814107 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.814083    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a6560b69-serving-cert\""
Sep 11 16:09:47.825735 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.825712    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"]
Sep 11 16:09:47.854412 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.854389    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:47.892672 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.892648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-openshift-service-ca\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.892782 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.892679    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-metrics-certs\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.892830 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.892795    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-maas-api-tls\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.892866 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.892831    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sm2x\" (UniqueName: \"kubernetes.io/projected/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-kube-api-access-7sm2x\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.956623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.956587    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:47.993586 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.993555    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-maas-api-tls\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.993716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.993594    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sm2x\" (UniqueName: \"kubernetes.io/projected/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-kube-api-access-7sm2x\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.993716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.993642    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-openshift-service-ca\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.993716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.993666    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-metrics-certs\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.994308 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.994283    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-openshift-service-ca\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.995982 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.995952    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-maas-api-tls\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:47.996060 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:47.996039    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-metrics-certs\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:48.004034 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:48.004010    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sm2x\" (UniqueName: \"kubernetes.io/projected/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-kube-api-access-7sm2x\") pod \"maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:48.060357 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:48.060336    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:48.124552 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:48.124529    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:48.398953 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:48.398929    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"]
Sep 11 16:09:48.400796 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:09:48.400769    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e200fe3_f36a_4d4c_8eed_d5c93a2647fc.slice/crio-b777bd5197c03d8cd14f790878dd29c98da69a3d8aeaaba9fc89b1da39d3bcbc WatchSource:0}: Error finding container b777bd5197c03d8cd14f790878dd29c98da69a3d8aeaaba9fc89b1da39d3bcbc: Status 404 returned error can't find the container with id b777bd5197c03d8cd14f790878dd29c98da69a3d8aeaaba9fc89b1da39d3bcbc
Sep 11 16:09:48.447874 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:48.447849    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw" event={"ID":"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc","Type":"ContainerStarted","Data":"b777bd5197c03d8cd14f790878dd29c98da69a3d8aeaaba9fc89b1da39d3bcbc"}
Sep 11 16:09:48.499813 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:48.499786    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:48.598883 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:48.598853    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:49.035372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.035303    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"]
Sep 11 16:09:49.039611 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.039589    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.048740 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.048713    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-9c5b09-serving-cert\""
Sep 11 16:09:49.057345 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.057310    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"]
Sep 11 16:09:49.093253 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.093219    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:49.104308 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.104285    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prwmc\" (UniqueName: \"kubernetes.io/projected/866fa52e-c898-4e77-9349-569acb6eee08-kube-api-access-prwmc\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.104393 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.104323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-maas-api-tls\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.104393 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.104352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-metrics-certs\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.104462 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.104408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/866fa52e-c898-4e77-9349-569acb6eee08-openshift-service-ca\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.192395 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.192367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:49.205550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.205526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prwmc\" (UniqueName: \"kubernetes.io/projected/866fa52e-c898-4e77-9349-569acb6eee08-kube-api-access-prwmc\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.205623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.205567    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-maas-api-tls\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.205704 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.205686    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-metrics-certs\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.205762 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:09:49.205734    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-serving-cert: secret "maas-api-e2e-autores-9c5b09-serving-cert" not found
Sep 11 16:09:49.205762 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.205741    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/866fa52e-c898-4e77-9349-569acb6eee08-openshift-service-ca\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.205862 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:09:49.205790    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-maas-api-tls podName:866fa52e-c898-4e77-9349-569acb6eee08 nodeName:}" failed. No retries permitted until 2026-09-11 16:09:49.705768873 +0000 UTC m=+883.423743972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-maas-api-tls") pod "maas-api-e2e-autores-9c5b09-6b5678846-dzrpc" (UID: "866fa52e-c898-4e77-9349-569acb6eee08") : secret "maas-api-e2e-autores-9c5b09-serving-cert" not found
Sep 11 16:09:49.206369 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.206350    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/866fa52e-c898-4e77-9349-569acb6eee08-openshift-service-ca\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.208047 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.208026    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-metrics-certs\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.214706 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.214688    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prwmc\" (UniqueName: \"kubernetes.io/projected/866fa52e-c898-4e77-9349-569acb6eee08-kube-api-access-prwmc\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.379298 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.379276    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:49.454512 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.454483    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw" event={"ID":"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc","Type":"ContainerStarted","Data":"0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba"}
Sep 11 16:09:49.454632 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.454590    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:49.474567 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.474527    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw" podStartSLOduration=2.184889382 podStartE2EDuration="2.474514175s" totalImagesPullingTime="289.624793ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:49.473550382 +0000 UTC m=+883.191525542" watchObservedRunningTime="2026-09-11 16:09:49.474514175 +0000 UTC m=+883.192489295"
Sep 11 16:09:49.624469 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.624443    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:49.710112 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.710044    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-maas-api-tls\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.712558 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.712533    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-maas-api-tls\") pod \"maas-api-e2e-autores-9c5b09-6b5678846-dzrpc\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:49.717949 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.717906    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:49.952547 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:49.952510    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:50.227269 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:50.227222    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"]
Sep 11 16:09:50.229789 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:09:50.229763    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod866fa52e_c898_4e77_9349_569acb6eee08.slice/crio-4279321202be46d271894f77c07ed7a3a9989af468e28c153a82cfa891a4922b WatchSource:0}: Error finding container 4279321202be46d271894f77c07ed7a3a9989af468e28c153a82cfa891a4922b: Status 404 returned error can't find the container with id 4279321202be46d271894f77c07ed7a3a9989af468e28c153a82cfa891a4922b
Sep 11 16:09:50.272928 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:50.272898    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:50.461371 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:50.461336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc" event={"ID":"866fa52e-c898-4e77-9349-569acb6eee08","Type":"ContainerStarted","Data":"4279321202be46d271894f77c07ed7a3a9989af468e28c153a82cfa891a4922b"}
Sep 11 16:09:51.038766 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:51.038731    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:51.468493 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:51.468457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc" event={"ID":"866fa52e-c898-4e77-9349-569acb6eee08","Type":"ContainerStarted","Data":"495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567"}
Sep 11 16:09:51.468935 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:51.468525    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:51.488253 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:51.488201    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc" podStartSLOduration=2.217523212 podStartE2EDuration="2.488185854s" totalImagesPullingTime="270.662642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:09:51.486387591 +0000 UTC m=+885.204362713" watchObservedRunningTime="2026-09-11 16:09:51.488185854 +0000 UTC m=+885.206160973"
Sep 11 16:09:51.700122 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:51.700063    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:52.007486 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:52.007448    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:52.140280 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:52.140235    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:52.310368 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:52.310339    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:52.463085 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:52.463048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:52.596599 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:52.596509    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:53.092298 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:53.092231    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:53.224769 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:53.224722    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:55.091752 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:55.091708    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:55.466662 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:55.466587    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:09:55.539812 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:55.539748    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:56.144302 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:56.144269    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:56.268300 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:56.268267    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:56.775304 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:56.775270    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:57.132179 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:57.132150    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:57.478852 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:57.478785    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:09:57.660339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:57.660310    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:58.685514 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:58.685476    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:59.720987 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.720911    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"]
Sep 11 16:09:59.724476 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.724456    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.727302 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.727271    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2b1bd5-serving-cert\""
Sep 11 16:09:59.734170 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.734133    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"]
Sep 11 16:09:59.795365 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.795339    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-maas-api-tls\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.795496 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.795440    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnggp\" (UniqueName: \"kubernetes.io/projected/97e7e7d0-1129-4dfd-8efa-74617a8dc086-kube-api-access-rnggp\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.795560 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.795503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-metrics-certs\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.795560 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.795539    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e7e7d0-1129-4dfd-8efa-74617a8dc086-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.820826 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.820805    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:59.895355 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.895326    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:09:59.896425 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.896402    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-maas-api-tls\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.896504 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.896461    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnggp\" (UniqueName: \"kubernetes.io/projected/97e7e7d0-1129-4dfd-8efa-74617a8dc086-kube-api-access-rnggp\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.896564 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.896515    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-metrics-certs\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.896629 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.896575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e7e7d0-1129-4dfd-8efa-74617a8dc086-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.896629 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:09:59.896582    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-serving-cert: secret "maas-api-e2e-inf-b-2b1bd5-serving-cert" not found
Sep 11 16:09:59.896732 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:09:59.896648    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-maas-api-tls podName:97e7e7d0-1129-4dfd-8efa-74617a8dc086 nodeName:}" failed. No retries permitted until 2026-09-11 16:10:00.396630597 +0000 UTC m=+894.114605702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-maas-api-tls") pod "maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c" (UID: "97e7e7d0-1129-4dfd-8efa-74617a8dc086") : secret "maas-api-e2e-inf-b-2b1bd5-serving-cert" not found
Sep 11 16:09:59.897270 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.897251    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e7e7d0-1129-4dfd-8efa-74617a8dc086-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.899049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.899026    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-metrics-certs\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:09:59.905367 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:09:59.905345    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnggp\" (UniqueName: \"kubernetes.io/projected/97e7e7d0-1129-4dfd-8efa-74617a8dc086-kube-api-access-rnggp\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:10:00.092147 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:00.092116    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:00.371490 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:00.371412    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:00.401456 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:00.401429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-maas-api-tls\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:10:00.403827 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:00.403793    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-maas-api-tls\") pod \"maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:10:00.468373 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:00.468340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:00.650398 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:00.650321    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:10:00.834512 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:00.834475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:00.924113 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:00.924087    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"]
Sep 11 16:10:00.926449 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:10:00.926425    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e7e7d0_1129_4dfd_8efa_74617a8dc086.slice/crio-7e10fa7fa07f9c92de871f97e935b6a9e6c1dda0132adb6b4031d5ace50800bb WatchSource:0}: Error finding container 7e10fa7fa07f9c92de871f97e935b6a9e6c1dda0132adb6b4031d5ace50800bb: Status 404 returned error can't find the container with id 7e10fa7fa07f9c92de871f97e935b6a9e6c1dda0132adb6b4031d5ace50800bb
Sep 11 16:10:01.151402 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:01.151372    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:01.519558 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:01.519483    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c" event={"ID":"97e7e7d0-1129-4dfd-8efa-74617a8dc086","Type":"ContainerStarted","Data":"5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1"}
Sep 11 16:10:01.519558 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:01.519533    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c" event={"ID":"97e7e7d0-1129-4dfd-8efa-74617a8dc086","Type":"ContainerStarted","Data":"7e10fa7fa07f9c92de871f97e935b6a9e6c1dda0132adb6b4031d5ace50800bb"}
Sep 11 16:10:01.519727 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:01.519576    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:10:01.543637 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:01.543589    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c" podStartSLOduration=2.234464437 podStartE2EDuration="2.54357585s" totalImagesPullingTime="309.111413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:01.540770987 +0000 UTC m=+895.258746126" watchObservedRunningTime="2026-09-11 16:10:01.54357585 +0000 UTC m=+895.261551004"
Sep 11 16:10:02.132101 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:02.132061    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:03.376038 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:03.375993    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:04.406375 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:04.406332    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:05.636363 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:05.636325    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:06.760677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:06.760647    2564 scope.go:117] "RemoveContainer" containerID="9b17782597a9b897e20fdf3b3b9f59b0a3bd229f5864296c556d33cdd1c8cc62"
Sep 11 16:10:07.529534 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:07.529505    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:10:09.031917 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:09.031885    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:10.577955 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:10.577924    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:11.223689 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.223653    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"]
Sep 11 16:10:11.223940 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.223886    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw" podUID="1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" containerName="maas-api" containerID="cri-o://0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba" gracePeriod=30
Sep 11 16:10:11.473625 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.473602    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:10:11.568783 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.568754    2564 generic.go:356] "Generic (PLEG): container finished" podID="1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" containerID="0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba" exitCode=0
Sep 11 16:10:11.568933 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.568815    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"
Sep 11 16:10:11.568933 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.568838    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw" event={"ID":"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc","Type":"ContainerDied","Data":"0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba"}
Sep 11 16:10:11.569012 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.568943    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw" event={"ID":"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc","Type":"ContainerDied","Data":"b777bd5197c03d8cd14f790878dd29c98da69a3d8aeaaba9fc89b1da39d3bcbc"}
Sep 11 16:10:11.569012 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.568982    2564 scope.go:117] "RemoveContainer" containerID="0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba"
Sep 11 16:10:11.579420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.579397    2564 scope.go:117] "RemoveContainer" containerID="0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba"
Sep 11 16:10:11.580159 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:10:11.580132    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba\": container with ID starting with 0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba not found: ID does not exist" containerID="0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba"
Sep 11 16:10:11.580362 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.580171    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba"} err="failed to get container status \"0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba\": rpc error: code = NotFound desc = could not find container \"0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba\": container with ID starting with 0820e2fd4eaea3b9a9bacd8a1feb32ce0b4c0ec018307603d60a2a8cee1330ba not found: ID does not exist"
Sep 11 16:10:11.590966 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.590942    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-metrics-certs\") pod \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") "
Sep 11 16:10:11.591089 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.591052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-maas-api-tls\") pod \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") "
Sep 11 16:10:11.591446 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.591425    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sm2x\" (UniqueName: \"kubernetes.io/projected/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-kube-api-access-7sm2x\") pod \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") "
Sep 11 16:10:11.591821 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.591544    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-openshift-service-ca\") pod \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\" (UID: \"1e200fe3-f36a-4d4c-8eed-d5c93a2647fc\") "
Sep 11 16:10:11.592060 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.592035    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" (UID: "1e200fe3-f36a-4d4c-8eed-d5c93a2647fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:10:11.595528 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.595498    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-kube-api-access-7sm2x" (OuterVolumeSpecName: "kube-api-access-7sm2x") pod "1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" (UID: "1e200fe3-f36a-4d4c-8eed-d5c93a2647fc"). InnerVolumeSpecName "kube-api-access-7sm2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:11.595632 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.595569    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" (UID: "1e200fe3-f36a-4d4c-8eed-d5c93a2647fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:10:11.595688 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.595658    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" (UID: "1e200fe3-f36a-4d4c-8eed-d5c93a2647fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:10:11.693276 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.693231    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:11.693276 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.693269    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:11.693276 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.693280    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sm2x\" (UniqueName: \"kubernetes.io/projected/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-kube-api-access-7sm2x\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:11.693510 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.693290    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:11.890006 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.889981    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"]
Sep 11 16:10:11.894201 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.894180    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a6560b69-86d4ff7b4b-2hvdw"]
Sep 11 16:10:11.984174 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:11.984148    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:12.182447 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:12.182363    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:12.288334 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:12.288301    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:12.410894 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:12.410861    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:12.702453 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:12.702422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:12.811593 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:12.811557    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" path="/var/lib/kubelet/pods/1e200fe3-f36a-4d4c-8eed-d5c93a2647fc/volumes"
Sep 11 16:10:12.906035 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:12.905996    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:13.234156 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:13.234101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:14.634300 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:14.634267    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:15.927176 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:15.927118    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:16.034820 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:16.034780    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:17.486938 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.486900    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"]
Sep 11 16:10:17.487306 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.487281    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" containerName="maas-api"
Sep 11 16:10:17.487306 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.487294    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" containerName="maas-api"
Sep 11 16:10:17.487394 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.487358    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e200fe3-f36a-4d4c-8eed-d5c93a2647fc" containerName="maas-api"
Sep 11 16:10:17.491359 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.491332    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.499367 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.499338    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-f8b85f0c-serving-cert\""
Sep 11 16:10:17.516289 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.516262    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"]
Sep 11 16:10:17.560131 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.560104    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:17.643338 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.643308    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-metrics-certs\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.643492 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.643354    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a7788-860f-458d-8b24-3513848593f0-openshift-service-ca\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.643492 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.643427    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfmb9\" (UniqueName: \"kubernetes.io/projected/e18a7788-860f-458d-8b24-3513848593f0-kube-api-access-dfmb9\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.643597 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.643526    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-maas-api-tls\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.665278 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.665250    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:17.744803 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.744735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-maas-api-tls\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.744803 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.744776    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-metrics-certs\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.744969 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.744802    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a7788-860f-458d-8b24-3513848593f0-openshift-service-ca\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.744969 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.744852    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfmb9\" (UniqueName: \"kubernetes.io/projected/e18a7788-860f-458d-8b24-3513848593f0-kube-api-access-dfmb9\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.744969 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:10:17.744874    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-serving-cert: secret "maas-api-e2e-migrate-f8b85f0c-serving-cert" not found
Sep 11 16:10:17.744969 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:10:17.744943    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-maas-api-tls podName:e18a7788-860f-458d-8b24-3513848593f0 nodeName:}" failed. No retries permitted until 2026-09-11 16:10:18.244927395 +0000 UTC m=+911.962902510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-maas-api-tls") pod "maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v" (UID: "e18a7788-860f-458d-8b24-3513848593f0") : secret "maas-api-e2e-migrate-f8b85f0c-serving-cert" not found
Sep 11 16:10:17.745559 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.745531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a7788-860f-458d-8b24-3513848593f0-openshift-service-ca\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.747226 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.747199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-metrics-certs\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:17.762444 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:17.762418    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfmb9\" (UniqueName: \"kubernetes.io/projected/e18a7788-860f-458d-8b24-3513848593f0-kube-api-access-dfmb9\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:18.058349 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.058279    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:18.249691 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.249660    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-maas-api-tls\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:18.251997 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.251969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-maas-api-tls\") pod \"maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:18.402612 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.402588    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:18.487727 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.487697    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:18.582452 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.582425    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:18.691868 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.691680    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"]
Sep 11 16:10:18.693555 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.693529    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"]
Sep 11 16:10:18.694201 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:10:18.694179    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode18a7788_860f_458d_8b24_3513848593f0.slice/crio-fb3c353491e10d8e62bf828284498a96777412313bb6e927d2fe64f7dd0a95b9 WatchSource:0}: Error finding container fb3c353491e10d8e62bf828284498a96777412313bb6e927d2fe64f7dd0a95b9: Status 404 returned error can't find the container with id fb3c353491e10d8e62bf828284498a96777412313bb6e927d2fe64f7dd0a95b9
Sep 11 16:10:18.698494 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.698474    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:18.701816 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.701797    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f8b85f0c\""
Sep 11 16:10:18.701906 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.701801    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f8b85f0c-dockercfg-2gqkk\""
Sep 11 16:10:18.711810 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.711791    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"]
Sep 11 16:10:18.855876 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.855838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskw7\" (UniqueName: \"kubernetes.io/projected/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-kube-api-access-bskw7\") pod \"payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq\" (UID: \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:18.856108 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.855897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq\" (UID: \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:18.956514 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.956430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskw7\" (UniqueName: \"kubernetes.io/projected/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-kube-api-access-bskw7\") pod \"payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq\" (UID: \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:18.956514 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.956479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq\" (UID: \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:18.957201 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.957174    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq\" (UID: \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:18.967960 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:18.967934    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskw7\" (UniqueName: \"kubernetes.io/projected/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-kube-api-access-bskw7\") pod \"payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq\" (UID: \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:19.008735 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.008715    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:19.141125 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.141097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:19.306711 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.306674    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"]
Sep 11 16:10:19.607158 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.607124    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq" event={"ID":"f71e2fbb-2a2a-49ad-8f73-d34042e4a592","Type":"ContainerStarted","Data":"ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f"}
Sep 11 16:10:19.607575 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.607177    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq" event={"ID":"f71e2fbb-2a2a-49ad-8f73-d34042e4a592","Type":"ContainerStarted","Data":"5e608bd07afe8c84272045f9d2281bb87c8207015296291001149ccceb01c9b8"}
Sep 11 16:10:19.607575 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.607266    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:19.608813 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.608789    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v" event={"ID":"e18a7788-860f-458d-8b24-3513848593f0","Type":"ContainerStarted","Data":"9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b"}
Sep 11 16:10:19.608875 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.608830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v" event={"ID":"e18a7788-860f-458d-8b24-3513848593f0","Type":"ContainerStarted","Data":"fb3c353491e10d8e62bf828284498a96777412313bb6e927d2fe64f7dd0a95b9"}
Sep 11 16:10:19.608919 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.608883    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:19.710354 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.710315    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq" podStartSLOduration=1.7103010250000001 podStartE2EDuration="1.710301025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:19.678982567 +0000 UTC m=+913.396957697" watchObservedRunningTime="2026-09-11 16:10:19.710301025 +0000 UTC m=+913.428276145"
Sep 11 16:10:19.737398 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:19.737355    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v" podStartSLOduration=2.281680606 podStartE2EDuration="2.73733796s" totalImagesPullingTime="455.657354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:19.734622449 +0000 UTC m=+913.452597587" watchObservedRunningTime="2026-09-11 16:10:19.73733796 +0000 UTC m=+913.455313084"
Sep 11 16:10:21.189906 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:21.189875    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:21.366442 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:21.366409    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:22.248311 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:22.248279    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:23.629701 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:23.629664    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:24.558010 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:24.557972    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:25.619059 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:25.619033    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:30.615522 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:30.615495    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:40.302636 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:40.302605    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:41.228589 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:41.228547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:42.187186 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:42.187153    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:43.127315 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:43.127259    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:44.087806 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:44.087776    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:45.111586 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:45.111550    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:45.305733 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:45.305702    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:45.427860 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:45.427788    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:45.554758 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:45.554724    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:46.331166 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:46.331107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:47.156763 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.156727    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"]
Sep 11 16:10:47.156997 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.156972    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v" podUID="e18a7788-860f-458d-8b24-3513848593f0" containerName="maas-api" containerID="cri-o://9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b" gracePeriod=30
Sep 11 16:10:47.208526 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.208475    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:47.307812 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.307779    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"]
Sep 11 16:10:47.308277 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.308013    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq" podUID="f71e2fbb-2a2a-49ad-8f73-d34042e4a592" containerName="payload-pre-processing" containerID="cri-o://ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f" gracePeriod=30
Sep 11 16:10:47.430535 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.430510    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:47.432363 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.432340    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:47.476082 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.476054    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-maas-api-tls\") pod \"e18a7788-860f-458d-8b24-3513848593f0\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") "
Sep 11 16:10:47.476218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.476108    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a7788-860f-458d-8b24-3513848593f0-openshift-service-ca\") pod \"e18a7788-860f-458d-8b24-3513848593f0\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") "
Sep 11 16:10:47.476218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.476201    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-metrics-certs\") pod \"e18a7788-860f-458d-8b24-3513848593f0\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") "
Sep 11 16:10:47.476369 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.476226    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfmb9\" (UniqueName: \"kubernetes.io/projected/e18a7788-860f-458d-8b24-3513848593f0-kube-api-access-dfmb9\") pod \"e18a7788-860f-458d-8b24-3513848593f0\" (UID: \"e18a7788-860f-458d-8b24-3513848593f0\") "
Sep 11 16:10:47.476672 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.476595    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e18a7788-860f-458d-8b24-3513848593f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e18a7788-860f-458d-8b24-3513848593f0" (UID: "e18a7788-860f-458d-8b24-3513848593f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:10:47.478659 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.478626    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e18a7788-860f-458d-8b24-3513848593f0" (UID: "e18a7788-860f-458d-8b24-3513848593f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:10:47.478771 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.478665    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e18a7788-860f-458d-8b24-3513848593f0" (UID: "e18a7788-860f-458d-8b24-3513848593f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:10:47.479571 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.479544    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18a7788-860f-458d-8b24-3513848593f0-kube-api-access-dfmb9" (OuterVolumeSpecName: "kube-api-access-dfmb9") pod "e18a7788-860f-458d-8b24-3513848593f0" (UID: "e18a7788-860f-458d-8b24-3513848593f0"). InnerVolumeSpecName "kube-api-access-dfmb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:47.555462 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.555440    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:47.577395 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.577371    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:47.577395 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.577392    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e18a7788-860f-458d-8b24-3513848593f0-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:47.577527 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.577401    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e18a7788-860f-458d-8b24-3513848593f0-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:47.577527 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.577411    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfmb9\" (UniqueName: \"kubernetes.io/projected/e18a7788-860f-458d-8b24-3513848593f0-kube-api-access-dfmb9\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:47.592631 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.592602    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:47.678561 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.678492    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bskw7\" (UniqueName: \"kubernetes.io/projected/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-kube-api-access-bskw7\") pod \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\" (UID: \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\") "
Sep 11 16:10:47.678561 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.678532    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-plugins-config-volume\") pod \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\" (UID: \"f71e2fbb-2a2a-49ad-8f73-d34042e4a592\") "
Sep 11 16:10:47.678939 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.678917    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f71e2fbb-2a2a-49ad-8f73-d34042e4a592" (UID: "f71e2fbb-2a2a-49ad-8f73-d34042e4a592"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:10:47.680606 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.680568    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-kube-api-access-bskw7" (OuterVolumeSpecName: "kube-api-access-bskw7") pod "f71e2fbb-2a2a-49ad-8f73-d34042e4a592" (UID: "f71e2fbb-2a2a-49ad-8f73-d34042e4a592"). InnerVolumeSpecName "kube-api-access-bskw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:10:47.739895 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.739859    2564 generic.go:356] "Generic (PLEG): container finished" podID="e18a7788-860f-458d-8b24-3513848593f0" containerID="9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b" exitCode=0
Sep 11 16:10:47.740036 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.739900    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"
Sep 11 16:10:47.740036 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.739947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v" event={"ID":"e18a7788-860f-458d-8b24-3513848593f0","Type":"ContainerDied","Data":"9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b"}
Sep 11 16:10:47.740036 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.739996    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v" event={"ID":"e18a7788-860f-458d-8b24-3513848593f0","Type":"ContainerDied","Data":"fb3c353491e10d8e62bf828284498a96777412313bb6e927d2fe64f7dd0a95b9"}
Sep 11 16:10:47.740036 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.740018    2564 scope.go:117] "RemoveContainer" containerID="9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b"
Sep 11 16:10:47.741531 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.741484    2564 generic.go:356] "Generic (PLEG): container finished" podID="f71e2fbb-2a2a-49ad-8f73-d34042e4a592" containerID="ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f" exitCode=0
Sep 11 16:10:47.741627 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.741534    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"
Sep 11 16:10:47.741627 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.741543    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq" event={"ID":"f71e2fbb-2a2a-49ad-8f73-d34042e4a592","Type":"ContainerDied","Data":"ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f"}
Sep 11 16:10:47.741627 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.741581    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq" event={"ID":"f71e2fbb-2a2a-49ad-8f73-d34042e4a592","Type":"ContainerDied","Data":"5e608bd07afe8c84272045f9d2281bb87c8207015296291001149ccceb01c9b8"}
Sep 11 16:10:47.749068 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.749052    2564 scope.go:117] "RemoveContainer" containerID="9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b"
Sep 11 16:10:47.749391 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:10:47.749374    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b\": container with ID starting with 9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b not found: ID does not exist" containerID="9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b"
Sep 11 16:10:47.749464 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.749397    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b"} err="failed to get container status \"9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b\": rpc error: code = NotFound desc = could not find container \"9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b\": container with ID starting with 9620fc18f79875e947200e3bb988e06af72858eeab15d9b46d427e640519076b not found: ID does not exist"
Sep 11 16:10:47.749464 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.749413    2564 scope.go:117] "RemoveContainer" containerID="ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f"
Sep 11 16:10:47.757456 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.757442    2564 scope.go:117] "RemoveContainer" containerID="ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f"
Sep 11 16:10:47.757694 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:10:47.757679    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f\": container with ID starting with ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f not found: ID does not exist" containerID="ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f"
Sep 11 16:10:47.757785 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.757699    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f"} err="failed to get container status \"ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f\": rpc error: code = NotFound desc = could not find container \"ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f\": container with ID starting with ccaf40220596499a04acaae9e5f71db3787bb37aeb52a4cac952a50e8033aa5f not found: ID does not exist"
Sep 11 16:10:47.779274 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.779232    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bskw7\" (UniqueName: \"kubernetes.io/projected/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-kube-api-access-bskw7\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:47.779274 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.779272    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f71e2fbb-2a2a-49ad-8f73-d34042e4a592-plugins-config-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:10:47.798197 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.795005    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"]
Sep 11 16:10:47.826724 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.826700    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-f8b85f0c-79d8b75f46-76n6v"]
Sep 11 16:10:47.836057 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.836036    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:47.851140 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.851118    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"]
Sep 11 16:10:47.857521 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.857501    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f8b85f0c-768c4bd85d-f8qgq"]
Sep 11 16:10:47.962911 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:47.962830    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:48.059537 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:48.059507    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:48.171530 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:48.171493    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:48.382227 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:48.382202    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:48.723457 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:48.723375    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:48.811318 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:48.811289    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e18a7788-860f-458d-8b24-3513848593f0" path="/var/lib/kubelet/pods/e18a7788-860f-458d-8b24-3513848593f0/volumes"
Sep 11 16:10:48.811649 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:48.811636    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71e2fbb-2a2a-49ad-8f73-d34042e4a592" path="/var/lib/kubelet/pods/f71e2fbb-2a2a-49ad-8f73-d34042e4a592/volumes"
Sep 11 16:10:49.036645 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:49.036564    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:49.232563 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:49.232533    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:49.885173 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:49.885138    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:50.046388 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:50.046347    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:50.488388 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:50.488358    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:51.253512 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:51.253477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:51.354058 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:51.354029    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:51.503089 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:51.503053    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:52.285450 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:52.285420    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:53.202582 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:53.202526    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:53.328990 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:53.328954    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:53.433171 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:53.433141    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:53.628405 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:53.628373    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:54.939592 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.939561    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"]
Sep 11 16:10:54.939959 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.939879    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71e2fbb-2a2a-49ad-8f73-d34042e4a592" containerName="payload-pre-processing"
Sep 11 16:10:54.939959 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.939889    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71e2fbb-2a2a-49ad-8f73-d34042e4a592" containerName="payload-pre-processing"
Sep 11 16:10:54.939959 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.939899    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e18a7788-860f-458d-8b24-3513848593f0" containerName="maas-api"
Sep 11 16:10:54.939959 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.939905    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e18a7788-860f-458d-8b24-3513848593f0" containerName="maas-api"
Sep 11 16:10:54.940087 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.939968    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71e2fbb-2a2a-49ad-8f73-d34042e4a592" containerName="payload-pre-processing"
Sep 11 16:10:54.940087 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.939980    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e18a7788-860f-458d-8b24-3513848593f0" containerName="maas-api"
Sep 11 16:10:54.944407 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.944388    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:54.948638 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.948619    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-20aaef10-serving-cert\""
Sep 11 16:10:54.974208 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:54.974184    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"]
Sep 11 16:10:55.005868 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.005846    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:55.039645 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.039621    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfzvb\" (UniqueName: \"kubernetes.io/projected/86d4e416-e41f-4ee2-92f1-2530d3b7e272-kube-api-access-jfzvb\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.039749 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.039663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-metrics-certs\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.039803 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.039762    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-maas-api-tls\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.039803 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.039794    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d4e416-e41f-4ee2-92f1-2530d3b7e272-openshift-service-ca\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.133926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.133896    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:55.141065 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.141043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-maas-api-tls\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.141132 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.141071    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d4e416-e41f-4ee2-92f1-2530d3b7e272-openshift-service-ca\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.141178 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.141133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfzvb\" (UniqueName: \"kubernetes.io/projected/86d4e416-e41f-4ee2-92f1-2530d3b7e272-kube-api-access-jfzvb\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.141178 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.141150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-metrics-certs\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.141775 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.141752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d4e416-e41f-4ee2-92f1-2530d3b7e272-openshift-service-ca\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.143371 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.143351    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-maas-api-tls\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.143489 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.143469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-metrics-certs\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.153731 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.153709    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfzvb\" (UniqueName: \"kubernetes.io/projected/86d4e416-e41f-4ee2-92f1-2530d3b7e272-kube-api-access-jfzvb\") pod \"maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.255480 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.255423    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:55.259834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.259809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:55.533665 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.533644    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"]
Sep 11 16:10:55.535763 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:10:55.535734    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d4e416_e41f_4ee2_92f1_2530d3b7e272.slice/crio-b52c936123643491af4cfefc140daad6fdcac4213dce18f93f44c6444c51d554 WatchSource:0}: Error finding container b52c936123643491af4cfefc140daad6fdcac4213dce18f93f44c6444c51d554: Status 404 returned error can't find the container with id b52c936123643491af4cfefc140daad6fdcac4213dce18f93f44c6444c51d554
Sep 11 16:10:55.717005 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.716955    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:55.780977 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.780943    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7" event={"ID":"86d4e416-e41f-4ee2-92f1-2530d3b7e272","Type":"ContainerStarted","Data":"b52c936123643491af4cfefc140daad6fdcac4213dce18f93f44c6444c51d554"}
Sep 11 16:10:55.868559 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.868536    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:55.916734 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.916710    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"]
Sep 11 16:10:55.921130 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.921114    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:10:55.924412 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.924392    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-20aaef10\""
Sep 11 16:10:55.924536 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.924472    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-20aaef10-dockercfg-mb6g2\""
Sep 11 16:10:55.931000 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:55.930979    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"]
Sep 11 16:10:56.051556 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.051527    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq72z\" (UniqueName: \"kubernetes.io/projected/da024e07-0dc5-425e-a4d0-f4df25e254d7-kube-api-access-kq72z\") pod \"payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7\" (UID: \"da024e07-0dc5-425e-a4d0-f4df25e254d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:10:56.051918 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.051580    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da024e07-0dc5-425e-a4d0-f4df25e254d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7\" (UID: \"da024e07-0dc5-425e-a4d0-f4df25e254d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:10:56.152130 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.152069    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq72z\" (UniqueName: \"kubernetes.io/projected/da024e07-0dc5-425e-a4d0-f4df25e254d7-kube-api-access-kq72z\") pod \"payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7\" (UID: \"da024e07-0dc5-425e-a4d0-f4df25e254d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:10:56.152130 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.152104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da024e07-0dc5-425e-a4d0-f4df25e254d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7\" (UID: \"da024e07-0dc5-425e-a4d0-f4df25e254d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:10:56.152818 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.152798    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da024e07-0dc5-425e-a4d0-f4df25e254d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7\" (UID: \"da024e07-0dc5-425e-a4d0-f4df25e254d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:10:56.161834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.161810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq72z\" (UniqueName: \"kubernetes.io/projected/da024e07-0dc5-425e-a4d0-f4df25e254d7-kube-api-access-kq72z\") pod \"payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7\" (UID: \"da024e07-0dc5-425e-a4d0-f4df25e254d7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:10:56.232802 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.232773    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:10:56.511757 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.511734    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"]
Sep 11 16:10:56.512938 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:10:56.512902    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda024e07_0dc5_425e_a4d0_f4df25e254d7.slice/crio-d1d8a9a5d58b8f00f2e32ffca52701192f1eaa83d73eb31d8b82870806dc2124 WatchSource:0}: Error finding container d1d8a9a5d58b8f00f2e32ffca52701192f1eaa83d73eb31d8b82870806dc2124: Status 404 returned error can't find the container with id d1d8a9a5d58b8f00f2e32ffca52701192f1eaa83d73eb31d8b82870806dc2124
Sep 11 16:10:56.787650 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.787552    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7" event={"ID":"86d4e416-e41f-4ee2-92f1-2530d3b7e272","Type":"ContainerStarted","Data":"c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489"}
Sep 11 16:10:56.787819 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.787727    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:10:56.789179 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.789150    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7" event={"ID":"da024e07-0dc5-425e-a4d0-f4df25e254d7","Type":"ContainerStarted","Data":"202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169"}
Sep 11 16:10:56.789352 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.789202    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7" event={"ID":"da024e07-0dc5-425e-a4d0-f4df25e254d7","Type":"ContainerStarted","Data":"d1d8a9a5d58b8f00f2e32ffca52701192f1eaa83d73eb31d8b82870806dc2124"}
Sep 11 16:10:56.789352 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.789277    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:10:56.852465 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.852418    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7" podStartSLOduration=1.85239902 podStartE2EDuration="1.85239902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:56.850229654 +0000 UTC m=+950.568204784" watchObservedRunningTime="2026-09-11 16:10:56.85239902 +0000 UTC m=+950.570374143"
Sep 11 16:10:56.852972 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:56.852943    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7" podStartSLOduration=2.393612083 podStartE2EDuration="2.852933299s" totalImagesPullingTime="459.321216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:10:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:10:56.82480996 +0000 UTC m=+950.542785079" watchObservedRunningTime="2026-09-11 16:10:56.852933299 +0000 UTC m=+950.570908420"
Sep 11 16:10:57.172084 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:57.172050    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:10:57.935032 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:10:57.935003    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:00.697602 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:00.697568    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:00.794341 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:00.794303    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:02.799011 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:02.798978    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:11:07.796323 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:07.796288    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:11:10.025899 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:10.025854    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:20.337920 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:20.337882    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:21.261214 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:21.261181    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:22.215793 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:22.215755    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:23.100568 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:23.100532    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:23.893908 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:23.893873    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:24.941275 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:24.941188    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:25.587877 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:25.587847    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:25.810488 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:25.810456    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:26.870040 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:26.870011    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:32.728114 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:32.728032    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:33.429268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.429221    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"]
Sep 11 16:11:33.429491 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.429466    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7" podUID="86d4e416-e41f-4ee2-92f1-2530d3b7e272" containerName="maas-api" containerID="cri-o://c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489" gracePeriod=30
Sep 11 16:11:33.469397 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.469362    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:33.561309 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.561277    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"]
Sep 11 16:11:33.561553 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.561523    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7" podUID="da024e07-0dc5-425e-a4d0-f4df25e254d7" containerName="payload-pre-processing" containerID="cri-o://202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169" gracePeriod=30
Sep 11 16:11:33.588121 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.588094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:33.685659 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.685634    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:11:33.689897 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.689871    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:33.750835 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.750805    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-maas-api-tls\") pod \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") "
Sep 11 16:11:33.751616 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.750916    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfzvb\" (UniqueName: \"kubernetes.io/projected/86d4e416-e41f-4ee2-92f1-2530d3b7e272-kube-api-access-jfzvb\") pod \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") "
Sep 11 16:11:33.751616 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.750945    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d4e416-e41f-4ee2-92f1-2530d3b7e272-openshift-service-ca\") pod \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") "
Sep 11 16:11:33.751616 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.751000    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-metrics-certs\") pod \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\" (UID: \"86d4e416-e41f-4ee2-92f1-2530d3b7e272\") "
Sep 11 16:11:33.752067 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.751980    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d4e416-e41f-4ee2-92f1-2530d3b7e272-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "86d4e416-e41f-4ee2-92f1-2530d3b7e272" (UID: "86d4e416-e41f-4ee2-92f1-2530d3b7e272"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:11:33.753579 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.753542    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "86d4e416-e41f-4ee2-92f1-2530d3b7e272" (UID: "86d4e416-e41f-4ee2-92f1-2530d3b7e272"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:11:33.754272 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.754160    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "86d4e416-e41f-4ee2-92f1-2530d3b7e272" (UID: "86d4e416-e41f-4ee2-92f1-2530d3b7e272"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:11:33.754674 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.754638    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d4e416-e41f-4ee2-92f1-2530d3b7e272-kube-api-access-jfzvb" (OuterVolumeSpecName: "kube-api-access-jfzvb") pod "86d4e416-e41f-4ee2-92f1-2530d3b7e272" (UID: "86d4e416-e41f-4ee2-92f1-2530d3b7e272"). InnerVolumeSpecName "kube-api-access-jfzvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:11:33.792248 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.792207    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:33.812114 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.812095    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:11:33.852089 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.852061    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da024e07-0dc5-425e-a4d0-f4df25e254d7-plugins-config-volume\") pod \"da024e07-0dc5-425e-a4d0-f4df25e254d7\" (UID: \"da024e07-0dc5-425e-a4d0-f4df25e254d7\") "
Sep 11 16:11:33.852262 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.852133    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq72z\" (UniqueName: \"kubernetes.io/projected/da024e07-0dc5-425e-a4d0-f4df25e254d7-kube-api-access-kq72z\") pod \"da024e07-0dc5-425e-a4d0-f4df25e254d7\" (UID: \"da024e07-0dc5-425e-a4d0-f4df25e254d7\") "
Sep 11 16:11:33.852409 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.852388    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:11:33.852502 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.852411    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/86d4e416-e41f-4ee2-92f1-2530d3b7e272-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:11:33.852502 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.852426    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfzvb\" (UniqueName: \"kubernetes.io/projected/86d4e416-e41f-4ee2-92f1-2530d3b7e272-kube-api-access-jfzvb\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:11:33.852502 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.852441    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/86d4e416-e41f-4ee2-92f1-2530d3b7e272-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:11:33.852622 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.852497    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da024e07-0dc5-425e-a4d0-f4df25e254d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da024e07-0dc5-425e-a4d0-f4df25e254d7" (UID: "da024e07-0dc5-425e-a4d0-f4df25e254d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:11:33.854494 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.854466    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da024e07-0dc5-425e-a4d0-f4df25e254d7-kube-api-access-kq72z" (OuterVolumeSpecName: "kube-api-access-kq72z") pod "da024e07-0dc5-425e-a4d0-f4df25e254d7" (UID: "da024e07-0dc5-425e-a4d0-f4df25e254d7"). InnerVolumeSpecName "kube-api-access-kq72z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:11:33.953167 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.953090    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da024e07-0dc5-425e-a4d0-f4df25e254d7-plugins-config-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:11:33.953167 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.953113    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq72z\" (UniqueName: \"kubernetes.io/projected/da024e07-0dc5-425e-a4d0-f4df25e254d7-kube-api-access-kq72z\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:11:33.959989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.959964    2564 generic.go:356] "Generic (PLEG): container finished" podID="86d4e416-e41f-4ee2-92f1-2530d3b7e272" containerID="c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489" exitCode=0
Sep 11 16:11:33.960114 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.960006    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"
Sep 11 16:11:33.960114 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.960059    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7" event={"ID":"86d4e416-e41f-4ee2-92f1-2530d3b7e272","Type":"ContainerDied","Data":"c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489"}
Sep 11 16:11:33.960250 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.960119    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7" event={"ID":"86d4e416-e41f-4ee2-92f1-2530d3b7e272","Type":"ContainerDied","Data":"b52c936123643491af4cfefc140daad6fdcac4213dce18f93f44c6444c51d554"}
Sep 11 16:11:33.960250 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.960155    2564 scope.go:117] "RemoveContainer" containerID="c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489"
Sep 11 16:11:33.961590 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.961565    2564 generic.go:356] "Generic (PLEG): container finished" podID="da024e07-0dc5-425e-a4d0-f4df25e254d7" containerID="202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169" exitCode=0
Sep 11 16:11:33.961692 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.961609    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"
Sep 11 16:11:33.961692 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.961621    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7" event={"ID":"da024e07-0dc5-425e-a4d0-f4df25e254d7","Type":"ContainerDied","Data":"202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169"}
Sep 11 16:11:33.961692 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.961658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7" event={"ID":"da024e07-0dc5-425e-a4d0-f4df25e254d7","Type":"ContainerDied","Data":"d1d8a9a5d58b8f00f2e32ffca52701192f1eaa83d73eb31d8b82870806dc2124"}
Sep 11 16:11:33.971323 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.971297    2564 scope.go:117] "RemoveContainer" containerID="c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489"
Sep 11 16:11:33.971712 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:11:33.971685    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489\": container with ID starting with c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489 not found: ID does not exist" containerID="c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489"
Sep 11 16:11:33.971792 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.971721    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489"} err="failed to get container status \"c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489\": rpc error: code = NotFound desc = could not find container \"c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489\": container with ID starting with c589746c6f60c9fe8f40a980bead4677a40b0f8982687b1bb6f44bc844301489 not found: ID does not exist"
Sep 11 16:11:33.971792 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.971744    2564 scope.go:117] "RemoveContainer" containerID="202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169"
Sep 11 16:11:33.979509 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.979491    2564 scope.go:117] "RemoveContainer" containerID="202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169"
Sep 11 16:11:33.979770 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:11:33.979753    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169\": container with ID starting with 202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169 not found: ID does not exist" containerID="202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169"
Sep 11 16:11:33.979819 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.979775    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169"} err="failed to get container status \"202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169\": rpc error: code = NotFound desc = could not find container \"202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169\": container with ID starting with 202b9ebc9230e8b6256d499fe8356b552e0a13feb0ea3dfc083274a468886169 not found: ID does not exist"
Sep 11 16:11:33.990149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:33.990123    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"]
Sep 11 16:11:34.003229 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:34.003208    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-20aaef10-6d495c6d89-hpwd7"]
Sep 11 16:11:34.015178 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:34.015157    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"]
Sep 11 16:11:34.020890 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:34.020871    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-20aaef10-8479b9d7c8-nz6t7"]
Sep 11 16:11:34.116282 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:34.116227    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:34.214855 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:34.214787    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:34.810730 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:34.810699    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86d4e416-e41f-4ee2-92f1-2530d3b7e272" path="/var/lib/kubelet/pods/86d4e416-e41f-4ee2-92f1-2530d3b7e272/volumes"
Sep 11 16:11:34.811066 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:34.811024    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da024e07-0dc5-425e-a4d0-f4df25e254d7" path="/var/lib/kubelet/pods/da024e07-0dc5-425e-a4d0-f4df25e254d7/volumes"
Sep 11 16:11:35.036731 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:35.036681    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:35.225681 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:35.225585    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:35.404392 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:35.404356    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:35.744632 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:35.744599    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:36.275132 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:36.275097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:36.380710 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:36.380680    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:37.172731 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:37.172701    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:38.476744 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:38.476709    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:38.627700 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:38.627608    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:39.438966 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:39.438930    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:40.299723 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:40.299693    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:41.194265 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:41.194220    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:42.276162 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.276136    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"]
Sep 11 16:11:42.276541 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.276511    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da024e07-0dc5-425e-a4d0-f4df25e254d7" containerName="payload-pre-processing"
Sep 11 16:11:42.276541 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.276537    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da024e07-0dc5-425e-a4d0-f4df25e254d7" containerName="payload-pre-processing"
Sep 11 16:11:42.276698 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.276573    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d4e416-e41f-4ee2-92f1-2530d3b7e272" containerName="maas-api"
Sep 11 16:11:42.276698 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.276581    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d4e416-e41f-4ee2-92f1-2530d3b7e272" containerName="maas-api"
Sep 11 16:11:42.276698 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.276648    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="da024e07-0dc5-425e-a4d0-f4df25e254d7" containerName="payload-pre-processing"
Sep 11 16:11:42.276698 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.276656    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d4e416-e41f-4ee2-92f1-2530d3b7e272" containerName="maas-api"
Sep 11 16:11:42.281219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.281203    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.287491 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.287468    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-455ccc2b-serving-cert\""
Sep 11 16:11:42.306371 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.306327    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"]
Sep 11 16:11:42.324212 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.324181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-maas-api-tls\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.324342 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.324262    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9d5\" (UniqueName: \"kubernetes.io/projected/9a38d6f1-08ba-49c1-b028-45beab680ff6-kube-api-access-kf9d5\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.324402 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.324352    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38d6f1-08ba-49c1-b028-45beab680ff6-openshift-service-ca\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.324402 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.324387    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-metrics-certs\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.365338 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.365308    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:42.425654 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.425623    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-maas-api-tls\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.425812 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.425682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9d5\" (UniqueName: \"kubernetes.io/projected/9a38d6f1-08ba-49c1-b028-45beab680ff6-kube-api-access-kf9d5\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.425812 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.425758    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38d6f1-08ba-49c1-b028-45beab680ff6-openshift-service-ca\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.425812 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:11:42.425765    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-serving-cert: secret "maas-api-e2e-derive-455ccc2b-serving-cert" not found
Sep 11 16:11:42.425812 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.425793    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-metrics-certs\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.426048 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:11:42.425826    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-maas-api-tls podName:9a38d6f1-08ba-49c1-b028-45beab680ff6 nodeName:}" failed. No retries permitted until 2026-09-11 16:11:42.925809312 +0000 UTC m=+996.643784414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-maas-api-tls") pod "maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6" (UID: "9a38d6f1-08ba-49c1-b028-45beab680ff6") : secret "maas-api-e2e-derive-455ccc2b-serving-cert" not found
Sep 11 16:11:42.426404 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.426378    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38d6f1-08ba-49c1-b028-45beab680ff6-openshift-service-ca\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.428224 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.428207    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-metrics-certs\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.439603 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.439583    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9d5\" (UniqueName: \"kubernetes.io/projected/9a38d6f1-08ba-49c1-b028-45beab680ff6-kube-api-access-kf9d5\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.486131 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.486106    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:42.702490 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.702457    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:42.930291 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.930257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-maas-api-tls\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.932434 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.932412    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-maas-api-tls\") pod \"maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:42.947812 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:42.947787    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:43.068339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.068297    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:43.093658 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.093637    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"]
Sep 11 16:11:43.096823 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.096806    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:11:43.099555 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.099534    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-455ccc2b-dockercfg-sx8l7\""
Sep 11 16:11:43.099654 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.099610    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-455ccc2b\""
Sep 11 16:11:43.113052 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.113032    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"]
Sep 11 16:11:43.132571 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.132551    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687e7c02-9cff-4a9b-9447-12432e5bc99f-plugins-config-volume\") pod \"payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9\" (UID: \"687e7c02-9cff-4a9b-9447-12432e5bc99f\") " pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:11:43.132687 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.132626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjs7\" (UniqueName: \"kubernetes.io/projected/687e7c02-9cff-4a9b-9447-12432e5bc99f-kube-api-access-vwjs7\") pod \"payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9\" (UID: \"687e7c02-9cff-4a9b-9447-12432e5bc99f\") " pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:11:43.190975 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.190954    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:43.234064 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.234028    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjs7\" (UniqueName: \"kubernetes.io/projected/687e7c02-9cff-4a9b-9447-12432e5bc99f-kube-api-access-vwjs7\") pod \"payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9\" (UID: \"687e7c02-9cff-4a9b-9447-12432e5bc99f\") " pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:11:43.234208 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.234111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687e7c02-9cff-4a9b-9447-12432e5bc99f-plugins-config-volume\") pod \"payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9\" (UID: \"687e7c02-9cff-4a9b-9447-12432e5bc99f\") " pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:11:43.234856 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.234831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687e7c02-9cff-4a9b-9447-12432e5bc99f-plugins-config-volume\") pod \"payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9\" (UID: \"687e7c02-9cff-4a9b-9447-12432e5bc99f\") " pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:11:43.245164 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.243823    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjs7\" (UniqueName: \"kubernetes.io/projected/687e7c02-9cff-4a9b-9447-12432e5bc99f-kube-api-access-vwjs7\") pod \"payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9\" (UID: \"687e7c02-9cff-4a9b-9447-12432e5bc99f\") " pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:11:43.407931 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.407858    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:11:43.495355 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.495324    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"]
Sep 11 16:11:43.497936 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:11:43.497907    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a38d6f1_08ba_49c1_b028_45beab680ff6.slice/crio-d9ac47fab2e4534970d58c72353f6ad97733782621fa2c46a322e14d59abf5d5 WatchSource:0}: Error finding container d9ac47fab2e4534970d58c72353f6ad97733782621fa2c46a322e14d59abf5d5: Status 404 returned error can't find the container with id d9ac47fab2e4534970d58c72353f6ad97733782621fa2c46a322e14d59abf5d5
Sep 11 16:11:43.499497 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.499473    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:11:43.692071 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:43.692052    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"]
Sep 11 16:11:43.694084 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:11:43.694060    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687e7c02_9cff_4a9b_9447_12432e5bc99f.slice/crio-fbeda4f09727153a6d092553f3570576592d9c792d810c26ef25732cf9639299 WatchSource:0}: Error finding container fbeda4f09727153a6d092553f3570576592d9c792d810c26ef25732cf9639299: Status 404 returned error can't find the container with id fbeda4f09727153a6d092553f3570576592d9c792d810c26ef25732cf9639299
Sep 11 16:11:44.012877 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:44.012848    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9" event={"ID":"687e7c02-9cff-4a9b-9447-12432e5bc99f","Type":"ContainerStarted","Data":"98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5"}
Sep 11 16:11:44.012989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:44.012910    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9" event={"ID":"687e7c02-9cff-4a9b-9447-12432e5bc99f","Type":"ContainerStarted","Data":"fbeda4f09727153a6d092553f3570576592d9c792d810c26ef25732cf9639299"}
Sep 11 16:11:44.012989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:44.012969    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:11:44.014607 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:44.014587    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6" event={"ID":"9a38d6f1-08ba-49c1-b028-45beab680ff6","Type":"ContainerStarted","Data":"d9ac47fab2e4534970d58c72353f6ad97733782621fa2c46a322e14d59abf5d5"}
Sep 11 16:11:44.038080 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:44.038031    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9" podStartSLOduration=1.038018341 podStartE2EDuration="1.038018341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:44.035403569 +0000 UTC m=+997.753378724" watchObservedRunningTime="2026-09-11 16:11:44.038018341 +0000 UTC m=+997.755993461"
Sep 11 16:11:45.020223 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:45.020184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6" event={"ID":"9a38d6f1-08ba-49c1-b028-45beab680ff6","Type":"ContainerStarted","Data":"93342c9c32ef841fc592d2dc7b23cad6575b1fd9122b045eb6286c7135628a67"}
Sep 11 16:11:45.020637 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:45.020354    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:45.041040 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:45.040996    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6" podStartSLOduration=2.5904493889999998 podStartE2EDuration="3.040980258s" totalImagesPullingTime="450.530869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:11:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:11:45.040528693 +0000 UTC m=+998.758503832" watchObservedRunningTime="2026-09-11 16:11:45.040980258 +0000 UTC m=+998.758955375"
Sep 11 16:11:51.035391 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:51.035363    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:11:53.165689 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:53.165652    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:11:55.021761 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:11:55.021733    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:12:06.551554 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:06.551514    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:07.347412 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:07.347379    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:08.237759 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:08.237732    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:08.734322 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:08.734291    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:09.197058 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:09.197022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:10.074262 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:10.074204    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:11.170490 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:11.170431    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:12.353459 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:12.353426    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:13.620570 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:13.620530    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:19.818367 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:19.818331    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:20.989095 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:20.989063    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"]
Sep 11 16:12:20.989510 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:20.989253    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6" podUID="9a38d6f1-08ba-49c1-b028-45beab680ff6" containerName="maas-api" containerID="cri-o://93342c9c32ef841fc592d2dc7b23cad6575b1fd9122b045eb6286c7135628a67" gracePeriod=30
Sep 11 16:12:21.030366 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.030329    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6" podUID="9a38d6f1-08ba-49c1-b028-45beab680ff6" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.40:8443/health\": dial tcp 10.134.0.40:8443: connect: connection refused"
Sep 11 16:12:21.096939 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.096914    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:21.191781 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.191739    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"]
Sep 11 16:12:21.192330 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.192298    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9" podUID="687e7c02-9cff-4a9b-9447-12432e5bc99f" containerName="payload-processing" containerID="cri-o://98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5" gracePeriod=30
Sep 11 16:12:21.204520 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.204492    2564 generic.go:356] "Generic (PLEG): container finished" podID="9a38d6f1-08ba-49c1-b028-45beab680ff6" containerID="93342c9c32ef841fc592d2dc7b23cad6575b1fd9122b045eb6286c7135628a67" exitCode=0
Sep 11 16:12:21.204762 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.204658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6" event={"ID":"9a38d6f1-08ba-49c1-b028-45beab680ff6","Type":"ContainerDied","Data":"93342c9c32ef841fc592d2dc7b23cad6575b1fd9122b045eb6286c7135628a67"}
Sep 11 16:12:21.248084 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.248025    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:21.321628 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.321606    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:12:21.357119 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.357074    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:21.431859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.431836    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf9d5\" (UniqueName: \"kubernetes.io/projected/9a38d6f1-08ba-49c1-b028-45beab680ff6-kube-api-access-kf9d5\") pod \"9a38d6f1-08ba-49c1-b028-45beab680ff6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") "
Sep 11 16:12:21.431994 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.431896    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-metrics-certs\") pod \"9a38d6f1-08ba-49c1-b028-45beab680ff6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") "
Sep 11 16:12:21.431994 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.431951    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38d6f1-08ba-49c1-b028-45beab680ff6-openshift-service-ca\") pod \"9a38d6f1-08ba-49c1-b028-45beab680ff6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") "
Sep 11 16:12:21.432104 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.432015    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-maas-api-tls\") pod \"9a38d6f1-08ba-49c1-b028-45beab680ff6\" (UID: \"9a38d6f1-08ba-49c1-b028-45beab680ff6\") "
Sep 11 16:12:21.432441 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.432416    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a38d6f1-08ba-49c1-b028-45beab680ff6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a38d6f1-08ba-49c1-b028-45beab680ff6" (UID: "9a38d6f1-08ba-49c1-b028-45beab680ff6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:12:21.433858 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.433827    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9a38d6f1-08ba-49c1-b028-45beab680ff6" (UID: "9a38d6f1-08ba-49c1-b028-45beab680ff6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:12:21.434002 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.433970    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a38d6f1-08ba-49c1-b028-45beab680ff6-kube-api-access-kf9d5" (OuterVolumeSpecName: "kube-api-access-kf9d5") pod "9a38d6f1-08ba-49c1-b028-45beab680ff6" (UID: "9a38d6f1-08ba-49c1-b028-45beab680ff6"). InnerVolumeSpecName "kube-api-access-kf9d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:12:21.434186 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.434165    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a38d6f1-08ba-49c1-b028-45beab680ff6" (UID: "9a38d6f1-08ba-49c1-b028-45beab680ff6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:12:21.441994 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.441972    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:12:21.447842 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.447822    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:21.532959 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.532893    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687e7c02-9cff-4a9b-9447-12432e5bc99f-plugins-config-volume\") pod \"687e7c02-9cff-4a9b-9447-12432e5bc99f\" (UID: \"687e7c02-9cff-4a9b-9447-12432e5bc99f\") "
Sep 11 16:12:21.533214 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.532970    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwjs7\" (UniqueName: \"kubernetes.io/projected/687e7c02-9cff-4a9b-9447-12432e5bc99f-kube-api-access-vwjs7\") pod \"687e7c02-9cff-4a9b-9447-12432e5bc99f\" (UID: \"687e7c02-9cff-4a9b-9447-12432e5bc99f\") "
Sep 11 16:12:21.533214 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.533115    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:21.533214 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.533131    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf9d5\" (UniqueName: \"kubernetes.io/projected/9a38d6f1-08ba-49c1-b028-45beab680ff6-kube-api-access-kf9d5\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:21.533214 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.533145    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a38d6f1-08ba-49c1-b028-45beab680ff6-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:21.533214 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.533159    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a38d6f1-08ba-49c1-b028-45beab680ff6-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:21.533671 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.533436    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687e7c02-9cff-4a9b-9447-12432e5bc99f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "687e7c02-9cff-4a9b-9447-12432e5bc99f" (UID: "687e7c02-9cff-4a9b-9447-12432e5bc99f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:12:21.534272 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.534231    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:21.535313 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.535289    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687e7c02-9cff-4a9b-9447-12432e5bc99f-kube-api-access-vwjs7" (OuterVolumeSpecName: "kube-api-access-vwjs7") pod "687e7c02-9cff-4a9b-9447-12432e5bc99f" (UID: "687e7c02-9cff-4a9b-9447-12432e5bc99f"). InnerVolumeSpecName "kube-api-access-vwjs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:12:21.624918 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.624877    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:21.634186 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.634164    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/687e7c02-9cff-4a9b-9447-12432e5bc99f-plugins-config-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:21.634186 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.634184    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwjs7\" (UniqueName: \"kubernetes.io/projected/687e7c02-9cff-4a9b-9447-12432e5bc99f-kube-api-access-vwjs7\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:21.912311 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.912279    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"]
Sep 11 16:12:21.912646 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.912625    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a38d6f1-08ba-49c1-b028-45beab680ff6" containerName="maas-api"
Sep 11 16:12:21.912646 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.912641    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a38d6f1-08ba-49c1-b028-45beab680ff6" containerName="maas-api"
Sep 11 16:12:21.912804 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.912662    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687e7c02-9cff-4a9b-9447-12432e5bc99f" containerName="payload-processing"
Sep 11 16:12:21.912804 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.912668    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="687e7c02-9cff-4a9b-9447-12432e5bc99f" containerName="payload-processing"
Sep 11 16:12:21.912804 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.912741    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a38d6f1-08ba-49c1-b028-45beab680ff6" containerName="maas-api"
Sep 11 16:12:21.912804 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.912750    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="687e7c02-9cff-4a9b-9447-12432e5bc99f" containerName="payload-processing"
Sep 11 16:12:21.914516 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.914484    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:21.915878 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.915861    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:21.919898 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.919881    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-497b3375-kserve-self-signed-certs\""
Sep 11 16:12:21.932642 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:21.932616    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"]
Sep 11 16:12:22.036683 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.036658    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lrb6\" (UniqueName: \"kubernetes.io/projected/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-kube-api-access-9lrb6\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.036948 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.036695    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-dshm\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.036948 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.036715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tls-certs\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.036948 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.036809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-model-cache\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.036948 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.036839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tmp-dir\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.036948 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.036893    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-home\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.119947 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.119916    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:22.138119 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.138096    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-model-cache\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.138219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.138124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tmp-dir\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.138219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.138156    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-home\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.138219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.138193    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lrb6\" (UniqueName: \"kubernetes.io/projected/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-kube-api-access-9lrb6\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.138219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.138216    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-dshm\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.138455 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.138232    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tls-certs\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.138594 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.138576    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-model-cache\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.138649 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.138605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-home\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.138649 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.138633    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tmp-dir\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.140338 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.140314    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-dshm\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.140650 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.140634    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tls-certs\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.158326 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.158308    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lrb6\" (UniqueName: \"kubernetes.io/projected/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-kube-api-access-9lrb6\") pod \"e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") " pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.211013 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.210957    2564 generic.go:356] "Generic (PLEG): container finished" podID="687e7c02-9cff-4a9b-9447-12432e5bc99f" containerID="98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5" exitCode=0
Sep 11 16:12:22.211013 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.211002    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"
Sep 11 16:12:22.211150 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.211046    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9" event={"ID":"687e7c02-9cff-4a9b-9447-12432e5bc99f","Type":"ContainerDied","Data":"98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5"}
Sep 11 16:12:22.211150 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.211093    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9" event={"ID":"687e7c02-9cff-4a9b-9447-12432e5bc99f","Type":"ContainerDied","Data":"fbeda4f09727153a6d092553f3570576592d9c792d810c26ef25732cf9639299"}
Sep 11 16:12:22.211150 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.211133    2564 scope.go:117] "RemoveContainer" containerID="98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5"
Sep 11 16:12:22.212569 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.212548    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6" event={"ID":"9a38d6f1-08ba-49c1-b028-45beab680ff6","Type":"ContainerDied","Data":"d9ac47fab2e4534970d58c72353f6ad97733782621fa2c46a322e14d59abf5d5"}
Sep 11 16:12:22.212569 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.212562    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"
Sep 11 16:12:22.220397 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.220215    2564 scope.go:117] "RemoveContainer" containerID="98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5"
Sep 11 16:12:22.220545 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:12:22.220525    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5\": container with ID starting with 98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5 not found: ID does not exist" containerID="98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5"
Sep 11 16:12:22.220604 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.220552    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5"} err="failed to get container status \"98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5\": rpc error: code = NotFound desc = could not find container \"98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5\": container with ID starting with 98db2e6edfb75dd459d97f4993bd0c45310b65b86420fbdf54817c3e9976a2f5 not found: ID does not exist"
Sep 11 16:12:22.220604 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.220567    2564 scope.go:117] "RemoveContainer" containerID="93342c9c32ef841fc592d2dc7b23cad6575b1fd9122b045eb6286c7135628a67"
Sep 11 16:12:22.225114 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.225098    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:22.290204 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.290180    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"]
Sep 11 16:12:22.291750 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.291727    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-455ccc2b-79476df5b9-n4js9"]
Sep 11 16:12:22.318356 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.318330    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"]
Sep 11 16:12:22.337523 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.337500    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-455ccc2b-6cdb65bf97-qcbl6"]
Sep 11 16:12:22.411015 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.410992    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:22.504102 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.504078    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"]
Sep 11 16:12:22.506723 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:12:22.506692    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f1fcfcc_e73c_4cbe_8bc1_70d124872797.slice/crio-7afdb2a9dafb6a3ebc21fa1a870c2e3766a5d0264b27e0ad359d43ee9d61ce05 WatchSource:0}: Error finding container 7afdb2a9dafb6a3ebc21fa1a870c2e3766a5d0264b27e0ad359d43ee9d61ce05: Status 404 returned error can't find the container with id 7afdb2a9dafb6a3ebc21fa1a870c2e3766a5d0264b27e0ad359d43ee9d61ce05
Sep 11 16:12:22.537897 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.537866    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:22.811086 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.811062    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687e7c02-9cff-4a9b-9447-12432e5bc99f" path="/var/lib/kubelet/pods/687e7c02-9cff-4a9b-9447-12432e5bc99f/volumes"
Sep 11 16:12:22.811507 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:22.811492    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a38d6f1-08ba-49c1-b028-45beab680ff6" path="/var/lib/kubelet/pods/9a38d6f1-08ba-49c1-b028-45beab680ff6/volumes"
Sep 11 16:12:23.145729 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:23.145625    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:23.220102 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:23.220073    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5" event={"ID":"6f1fcfcc-e73c-4cbe-8bc1-70d124872797","Type":"ContainerStarted","Data":"233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201"}
Sep 11 16:12:23.220234 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:23.220107    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5" event={"ID":"6f1fcfcc-e73c-4cbe-8bc1-70d124872797","Type":"ContainerStarted","Data":"7afdb2a9dafb6a3ebc21fa1a870c2e3766a5d0264b27e0ad359d43ee9d61ce05"}
Sep 11 16:12:23.992367 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:23.992336    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:24.531464 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:24.531431    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:25.424057 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:25.424027    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:25.566444 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:25.566412    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:26.714628 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:26.714592    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:27.608560 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:27.608528    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:28.714102 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:28.714069    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:30.945794 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:30.945726    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:31.052587 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:31.052556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:31.156048 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:31.156018    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:31.255566 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:31.255484    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:31.697617 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:31.697582    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:32.226209 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:32.226181    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:32.226626 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:32.226219    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:32.238321 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:32.238299    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:32.273497 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:32.273473    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:32.304781 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:32.304741    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5" podStartSLOduration=11.304728388000001 podStartE2EDuration="11.304728388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:12:23.268841319 +0000 UTC m=+1036.986816438" watchObservedRunningTime="2026-09-11 16:12:32.304728388 +0000 UTC m=+1046.022703507"
Sep 11 16:12:35.555685 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.555592    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"]
Sep 11 16:12:35.556140 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.555821    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc" podUID="866fa52e-c898-4e77-9349-569acb6eee08" containerName="maas-api" containerID="cri-o://495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567" gracePeriod=30
Sep 11 16:12:35.591339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.591292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:35.690423 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.690392    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:35.790758 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.790727    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:35.806749 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.806695    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:12:35.890717 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.890683    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:35.947823 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.947797    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/866fa52e-c898-4e77-9349-569acb6eee08-openshift-service-ca\") pod \"866fa52e-c898-4e77-9349-569acb6eee08\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") "
Sep 11 16:12:35.947976 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.947853    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-maas-api-tls\") pod \"866fa52e-c898-4e77-9349-569acb6eee08\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") "
Sep 11 16:12:35.947976 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.947873    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-metrics-certs\") pod \"866fa52e-c898-4e77-9349-569acb6eee08\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") "
Sep 11 16:12:35.947976 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.947901    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prwmc\" (UniqueName: \"kubernetes.io/projected/866fa52e-c898-4e77-9349-569acb6eee08-kube-api-access-prwmc\") pod \"866fa52e-c898-4e77-9349-569acb6eee08\" (UID: \"866fa52e-c898-4e77-9349-569acb6eee08\") "
Sep 11 16:12:35.948277 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.948252    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/866fa52e-c898-4e77-9349-569acb6eee08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "866fa52e-c898-4e77-9349-569acb6eee08" (UID: "866fa52e-c898-4e77-9349-569acb6eee08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:12:35.950044 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.950021    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "866fa52e-c898-4e77-9349-569acb6eee08" (UID: "866fa52e-c898-4e77-9349-569acb6eee08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:12:35.950111 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.950048    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "866fa52e-c898-4e77-9349-569acb6eee08" (UID: "866fa52e-c898-4e77-9349-569acb6eee08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:12:35.950339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:35.950320    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/866fa52e-c898-4e77-9349-569acb6eee08-kube-api-access-prwmc" (OuterVolumeSpecName: "kube-api-access-prwmc") pod "866fa52e-c898-4e77-9349-569acb6eee08" (UID: "866fa52e-c898-4e77-9349-569acb6eee08"). InnerVolumeSpecName "kube-api-access-prwmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:12:36.048792 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.048762    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/866fa52e-c898-4e77-9349-569acb6eee08-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:36.048792 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.048791    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:36.048792 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.048799    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/866fa52e-c898-4e77-9349-569acb6eee08-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:36.048981 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.048807    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prwmc\" (UniqueName: \"kubernetes.io/projected/866fa52e-c898-4e77-9349-569acb6eee08-kube-api-access-prwmc\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:36.179014 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.178986    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:36.281472 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.281443    2564 generic.go:356] "Generic (PLEG): container finished" podID="866fa52e-c898-4e77-9349-569acb6eee08" containerID="495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567" exitCode=0
Sep 11 16:12:36.281617 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.281489    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"
Sep 11 16:12:36.281617 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.281491    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc" event={"ID":"866fa52e-c898-4e77-9349-569acb6eee08","Type":"ContainerDied","Data":"495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567"}
Sep 11 16:12:36.281617 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.281563    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc" event={"ID":"866fa52e-c898-4e77-9349-569acb6eee08","Type":"ContainerDied","Data":"4279321202be46d271894f77c07ed7a3a9989af468e28c153a82cfa891a4922b"}
Sep 11 16:12:36.281617 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.281598    2564 scope.go:117] "RemoveContainer" containerID="495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567"
Sep 11 16:12:36.291888 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.291869    2564 scope.go:117] "RemoveContainer" containerID="495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567"
Sep 11 16:12:36.292360 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:12:36.292291    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567\": container with ID starting with 495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567 not found: ID does not exist" containerID="495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567"
Sep 11 16:12:36.292360 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.292324    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567"} err="failed to get container status \"495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567\": rpc error: code = NotFound desc = could not find container \"495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567\": container with ID starting with 495afe5dca64a98f9634f370f3bc6d6f0576ba4d205b30ef970be31ba6a9b567 not found: ID does not exist"
Sep 11 16:12:36.302975 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.302924    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:36.304300 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.304282    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"]
Sep 11 16:12:36.310166 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.310147    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-9c5b09-6b5678846-dzrpc"]
Sep 11 16:12:36.810113 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:36.810081    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="866fa52e-c898-4e77-9349-569acb6eee08" path="/var/lib/kubelet/pods/866fa52e-c898-4e77-9349-569acb6eee08/volumes"
Sep 11 16:12:37.290788 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:37.290758    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:38.241924 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:38.241894    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:41.223111 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:41.223078    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:42.328433 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:42.328402    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"]
Sep 11 16:12:42.352321 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:42.352284    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:42.667090 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:42.667010    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:42.762961 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:42.762933    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:42.945514 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:42.945428    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:43.390865 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:43.390829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:43.873385 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:43.873351    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:44.166704 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:44.166621    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:47.520129 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:47.520097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:47.651611 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:47.651580    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:48.889899 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:48.889862    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:49.013900 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:49.013864    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:50.236066 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.236029    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"]
Sep 11 16:12:50.236522 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.236397    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="866fa52e-c898-4e77-9349-569acb6eee08" containerName="maas-api"
Sep 11 16:12:50.236522 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.236409    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="866fa52e-c898-4e77-9349-569acb6eee08" containerName="maas-api"
Sep 11 16:12:50.236522 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.236463    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="866fa52e-c898-4e77-9349-569acb6eee08" containerName="maas-api"
Sep 11 16:12:50.240352 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.240328    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.243637 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.243616    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-dacdbe-serving-cert\""
Sep 11 16:12:50.247103 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.247054    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-maas-api-tls\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.247103 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.247090    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x7r4\" (UniqueName: \"kubernetes.io/projected/42addc5a-6fe2-4c84-a422-975fd8589f6b-kube-api-access-5x7r4\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.247201 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.247164    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42addc5a-6fe2-4c84-a422-975fd8589f6b-openshift-service-ca\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.247305 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.247285    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-metrics-certs\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.256911 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.256889    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"]
Sep 11 16:12:50.290208 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.290185    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:50.347818 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.347790    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42addc5a-6fe2-4c84-a422-975fd8589f6b-openshift-service-ca\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.347948 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.347862    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-metrics-certs\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.347948 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.347915    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-maas-api-tls\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.348151 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:12:50.348017    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-serving-cert: secret "maas-api-e2e-explicit-dacdbe-serving-cert" not found
Sep 11 16:12:50.348151 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:12:50.348075    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-maas-api-tls podName:42addc5a-6fe2-4c84-a422-975fd8589f6b nodeName:}" failed. No retries permitted until 2026-09-11 16:12:50.84805559 +0000 UTC m=+1064.566030707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-maas-api-tls") pod "maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf" (UID: "42addc5a-6fe2-4c84-a422-975fd8589f6b") : secret "maas-api-e2e-explicit-dacdbe-serving-cert" not found
Sep 11 16:12:50.348151 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.348101    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x7r4\" (UniqueName: \"kubernetes.io/projected/42addc5a-6fe2-4c84-a422-975fd8589f6b-kube-api-access-5x7r4\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.348542 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.348521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42addc5a-6fe2-4c84-a422-975fd8589f6b-openshift-service-ca\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.350804 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.350786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-metrics-certs\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.362983 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.362957    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x7r4\" (UniqueName: \"kubernetes.io/projected/42addc5a-6fe2-4c84-a422-975fd8589f6b-kube-api-access-5x7r4\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.385839 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.385805    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:50.762771 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.762735    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:50.852377 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.852343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-maas-api-tls\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.854675 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.854645    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-maas-api-tls\") pod \"maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:50.926178 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:50.926144    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:51.036862 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.036790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:51.135206 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.135178    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"]
Sep 11 16:12:51.138650 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.138635    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:12:51.141877 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.141852    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-dacdbe\""
Sep 11 16:12:51.142095 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.142080    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-dacdbe-dockercfg-hxh4g\""
Sep 11 16:12:51.150406 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.150389    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:51.154789 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.154759    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs\" (UID: \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:12:51.154959 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.154942    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxqmb\" (UniqueName: \"kubernetes.io/projected/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-kube-api-access-sxqmb\") pod \"payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs\" (UID: \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:12:51.157233 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.157215    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"]
Sep 11 16:12:51.256049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.256014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs\" (UID: \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:12:51.256439 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.256073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxqmb\" (UniqueName: \"kubernetes.io/projected/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-kube-api-access-sxqmb\") pod \"payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs\" (UID: \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:12:51.256747 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.256727    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs\" (UID: \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:12:51.267786 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.267761    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxqmb\" (UniqueName: \"kubernetes.io/projected/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-kube-api-access-sxqmb\") pod \"payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs\" (UID: \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:12:51.428160 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.428051    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"]
Sep 11 16:12:51.430321 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:12:51.430297    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42addc5a_6fe2_4c84_a422_975fd8589f6b.slice/crio-0eac6943955c7c95fffe5ca1f529c658b6e8852af50ef20170b88497fd99da19 WatchSource:0}: Error finding container 0eac6943955c7c95fffe5ca1f529c658b6e8852af50ef20170b88497fd99da19: Status 404 returned error can't find the container with id 0eac6943955c7c95fffe5ca1f529c658b6e8852af50ef20170b88497fd99da19
Sep 11 16:12:51.448877 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.448846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:12:51.725828 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:51.725760    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"]
Sep 11 16:12:51.728867 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:12:51.728839    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7ef2ef_f84b_473c_91a9_4e6d5ecdce8e.slice/crio-dd36f1de1d92b73e236a7c39518ebf58a164e3f325e6903bbd59e8522fdc1a18 WatchSource:0}: Error finding container dd36f1de1d92b73e236a7c39518ebf58a164e3f325e6903bbd59e8522fdc1a18: Status 404 returned error can't find the container with id dd36f1de1d92b73e236a7c39518ebf58a164e3f325e6903bbd59e8522fdc1a18
Sep 11 16:12:52.361949 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:52.361857    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf" event={"ID":"42addc5a-6fe2-4c84-a422-975fd8589f6b","Type":"ContainerStarted","Data":"1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1"}
Sep 11 16:12:52.361949 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:52.361919    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf" event={"ID":"42addc5a-6fe2-4c84-a422-975fd8589f6b","Type":"ContainerStarted","Data":"0eac6943955c7c95fffe5ca1f529c658b6e8852af50ef20170b88497fd99da19"}
Sep 11 16:12:52.361949 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:52.361950    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:52.363687 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:52.363658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs" event={"ID":"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e","Type":"ContainerStarted","Data":"35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515"}
Sep 11 16:12:52.363828 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:52.363708    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs" event={"ID":"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e","Type":"ContainerStarted","Data":"dd36f1de1d92b73e236a7c39518ebf58a164e3f325e6903bbd59e8522fdc1a18"}
Sep 11 16:12:52.384995 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:52.384952    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf" podStartSLOduration=1.748675609 podStartE2EDuration="2.384937429s" totalImagesPullingTime="636.26182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:12:52.382880317 +0000 UTC m=+1066.100855436" watchObservedRunningTime="2026-09-11 16:12:52.384937429 +0000 UTC m=+1066.102912552"
Sep 11 16:12:52.404168 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:52.404127    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs" podStartSLOduration=1.404113248 podStartE2EDuration="1.404113248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:12:52.403463499 +0000 UTC m=+1066.121438640" watchObservedRunningTime="2026-09-11 16:12:52.404113248 +0000 UTC m=+1066.122088368"
Sep 11 16:12:53.368781 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:53.368751    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:12:57.348394 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.348352    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5" podUID="6f1fcfcc-e73c-4cbe-8bc1-70d124872797" containerName="main" containerID="cri-o://233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201" gracePeriod=45
Sep 11 16:12:57.603681 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.603632    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:57.613349 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.613322    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:12:57.707860 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.707840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tls-certs\") pod \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") "
Sep 11 16:12:57.707982 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.707873    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-model-cache\") pod \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") "
Sep 11 16:12:57.707982 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.707891    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-home\") pod \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") "
Sep 11 16:12:57.707982 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.707941    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lrb6\" (UniqueName: \"kubernetes.io/projected/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-kube-api-access-9lrb6\") pod \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") "
Sep 11 16:12:57.707982 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.707981    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tmp-dir\") pod \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") "
Sep 11 16:12:57.708195 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.708026    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-dshm\") pod \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\" (UID: \"6f1fcfcc-e73c-4cbe-8bc1-70d124872797\") "
Sep 11 16:12:57.708195 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.708123    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f1fcfcc-e73c-4cbe-8bc1-70d124872797" (UID: "6f1fcfcc-e73c-4cbe-8bc1-70d124872797"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:12:57.708319 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.708198    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-home" (OuterVolumeSpecName: "home") pod "6f1fcfcc-e73c-4cbe-8bc1-70d124872797" (UID: "6f1fcfcc-e73c-4cbe-8bc1-70d124872797"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:12:57.708319 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.708237    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f1fcfcc-e73c-4cbe-8bc1-70d124872797" (UID: "6f1fcfcc-e73c-4cbe-8bc1-70d124872797"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:12:57.708386 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.708325    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tmp-dir\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:57.708386 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.708341    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-model-cache\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:57.708386 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.708351    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-home\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:57.710114 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.710092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-dshm" (OuterVolumeSpecName: "dshm") pod "6f1fcfcc-e73c-4cbe-8bc1-70d124872797" (UID: "6f1fcfcc-e73c-4cbe-8bc1-70d124872797"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:12:57.710409 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.710392    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f1fcfcc-e73c-4cbe-8bc1-70d124872797" (UID: "6f1fcfcc-e73c-4cbe-8bc1-70d124872797"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:12:57.710409 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.710395    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-kube-api-access-9lrb6" (OuterVolumeSpecName: "kube-api-access-9lrb6") pod "6f1fcfcc-e73c-4cbe-8bc1-70d124872797" (UID: "6f1fcfcc-e73c-4cbe-8bc1-70d124872797"). InnerVolumeSpecName "kube-api-access-9lrb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:12:57.809418 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.809397    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-dshm\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:57.809418 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.809419    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-tls-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:57.809534 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:57.809430    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lrb6\" (UniqueName: \"kubernetes.io/projected/6f1fcfcc-e73c-4cbe-8bc1-70d124872797-kube-api-access-9lrb6\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:12:58.373774 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.373752    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:12:58.392823 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.392797    2564 generic.go:356] "Generic (PLEG): container finished" podID="6f1fcfcc-e73c-4cbe-8bc1-70d124872797" containerID="233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201" exitCode=0
Sep 11 16:12:58.392944 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.392841    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"
Sep 11 16:12:58.392944 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.392881    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5" event={"ID":"6f1fcfcc-e73c-4cbe-8bc1-70d124872797","Type":"ContainerDied","Data":"233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201"}
Sep 11 16:12:58.392944 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.392923    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5" event={"ID":"6f1fcfcc-e73c-4cbe-8bc1-70d124872797","Type":"ContainerDied","Data":"7afdb2a9dafb6a3ebc21fa1a870c2e3766a5d0264b27e0ad359d43ee9d61ce05"}
Sep 11 16:12:58.392944 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.392940    2564 scope.go:117] "RemoveContainer" containerID="233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201"
Sep 11 16:12:58.401581 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.401567    2564 scope.go:117] "RemoveContainer" containerID="233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201"
Sep 11 16:12:58.401836 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:12:58.401816    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201\": container with ID starting with 233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201 not found: ID does not exist" containerID="233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201"
Sep 11 16:12:58.401924 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.401847    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201"} err="failed to get container status \"233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201\": rpc error: code = NotFound desc = could not find container \"233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201\": container with ID starting with 233230e47ebdf6189dad8b9b771e67df333f879d4ce4a8e005baf30a73f1b201 not found: ID does not exist"
Sep 11 16:12:58.426641 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.426616    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"]
Sep 11 16:12:58.430318 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.430298    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-497b3375-kserve-58c6cf5cc9-skgg5"]
Sep 11 16:12:58.810100 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:12:58.810022    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f1fcfcc-e73c-4cbe-8bc1-70d124872797" path="/var/lib/kubelet/pods/6f1fcfcc-e73c-4cbe-8bc1-70d124872797/volumes"
Sep 11 16:13:03.893258 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:03.893164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:04.040740 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:04.040706    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:04.190291 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:04.190196    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:04.376610 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:04.376584    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:13:07.263655 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:07.263575    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:07.336323 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:07.336296    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:12.827495 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:12.827464    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:13.754413 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:13.754374    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:14.564622 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:14.564586    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:15.419030 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:15.418999    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:17.500223 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:17.500192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:19.995297 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:19.995264    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:26.548501 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:26.548468    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:26.650150 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:26.650120    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:26.754323 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:26.754294    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:37.210194 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:37.210164    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:37.899654 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:37.899618    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:37.999944 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:37.999916    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:42.207053 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.207024    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"]
Sep 11 16:13:42.207438 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.207383    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f1fcfcc-e73c-4cbe-8bc1-70d124872797" containerName="main"
Sep 11 16:13:42.207438 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.207396    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f1fcfcc-e73c-4cbe-8bc1-70d124872797" containerName="main"
Sep 11 16:13:42.207510 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.207455    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f1fcfcc-e73c-4cbe-8bc1-70d124872797" containerName="main"
Sep 11 16:13:42.210497 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.210479    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.213709 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.213687    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-dacdbe\"/\"default-dockercfg-vmlrv\""
Sep 11 16:13:42.214946 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.214924    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-dacdbe\"/\"kube-root-ca.crt\""
Sep 11 16:13:42.215050 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.214996    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-dacdbe\"/\"openshift-service-ca.crt\""
Sep 11 16:13:42.215113 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.215054    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-dacdbe\"/\"explicit-model-dacdbe-kserve-self-signed-certs\""
Sep 11 16:13:42.226908 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.226889    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"]
Sep 11 16:13:42.236126 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.236107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:42.347290 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.347261    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-home\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.347290 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.347294    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k22f\" (UniqueName: \"kubernetes.io/projected/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-kube-api-access-9k22f\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.347533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.347324    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tls-certs\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.347533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.347441    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-dshm\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.347533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.347478    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-model-cache\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.347533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.347511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tmp-dir\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.448075 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.448044    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-dshm\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.448075 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.448074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-model-cache\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.448286 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.448102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tmp-dir\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.448286 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.448173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-home\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.448286 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.448198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k22f\" (UniqueName: \"kubernetes.io/projected/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-kube-api-access-9k22f\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.448286 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.448229    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tls-certs\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.448549 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.448524    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-model-cache\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.448637 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.448596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tmp-dir\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.448698 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.448637    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-home\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.450517 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.450487    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-dshm\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.450640 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.450625    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tls-certs\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.459701 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.459641    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k22f\" (UniqueName: \"kubernetes.io/projected/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-kube-api-access-9k22f\") pod \"explicit-model-dacdbe-kserve-d94f77d7f-7nl2b\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") " pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.519586 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.519562    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:42.796847 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:42.796817    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"]
Sep 11 16:13:42.798305 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:13:42.798279    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53b20f6a_a22b_4dbc_bf62_6c606bd49b35.slice/crio-caa0b4aaa3fb440d13d9bde313080bd21574aee5fe5be1e9262ab10e13ac4d89 WatchSource:0}: Error finding container caa0b4aaa3fb440d13d9bde313080bd21574aee5fe5be1e9262ab10e13ac4d89: Status 404 returned error can't find the container with id caa0b4aaa3fb440d13d9bde313080bd21574aee5fe5be1e9262ab10e13ac4d89
Sep 11 16:13:43.607312 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:43.607277    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b" event={"ID":"53b20f6a-a22b-4dbc-bf62-6c606bd49b35","Type":"ContainerStarted","Data":"d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b"}
Sep 11 16:13:43.607795 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:43.607318    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b" event={"ID":"53b20f6a-a22b-4dbc-bf62-6c606bd49b35","Type":"ContainerStarted","Data":"caa0b4aaa3fb440d13d9bde313080bd21574aee5fe5be1e9262ab10e13ac4d89"}
Sep 11 16:13:43.635038 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:43.634987    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b" podStartSLOduration=1.634967442 podStartE2EDuration="1.634967442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:13:43.634083281 +0000 UTC m=+1117.352058426" watchObservedRunningTime="2026-09-11 16:13:43.634967442 +0000 UTC m=+1117.352942563"
Sep 11 16:13:44.904787 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:44.904756    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"]
Sep 11 16:13:44.913456 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:44.913422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:45.054683 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:45.054650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:45.158731 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:45.158641    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:45.262530 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:45.262486    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:45.976745 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:45.976697    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:46.104619 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:46.104587    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:46.508888 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:46.508856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:47.684336 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:47.684303    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:47.779854 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:47.779828    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:48.552401 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:48.552367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:49.375945 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:49.375913    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:52.520699 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:52.520663    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:13:54.126000 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:54.125956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:54.211262 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:54.211211    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:54.779256 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:54.779211    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:55.803307 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:55.803264    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"]
Sep 11 16:13:55.803725 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:55.803484    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf" podUID="42addc5a-6fe2-4c84-a422-975fd8589f6b" containerName="maas-api" containerID="cri-o://1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1" gracePeriod=30
Sep 11 16:13:55.845967 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:55.845938    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:55.904015 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:55.903981    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"]
Sep 11 16:13:55.904305 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:55.904256    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs" podUID="6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e" containerName="payload-pre-processing" containerID="cri-o://35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515" gracePeriod=30
Sep 11 16:13:55.945729 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:55.945696    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:56.044936 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.044907    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:56.065540 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.065518    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:13:56.065801 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.065779    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-metrics-certs\") pod \"42addc5a-6fe2-4c84-a422-975fd8589f6b\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") "
Sep 11 16:13:56.065900 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.065853    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5x7r4\" (UniqueName: \"kubernetes.io/projected/42addc5a-6fe2-4c84-a422-975fd8589f6b-kube-api-access-5x7r4\") pod \"42addc5a-6fe2-4c84-a422-975fd8589f6b\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") "
Sep 11 16:13:56.065959 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.065905    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42addc5a-6fe2-4c84-a422-975fd8589f6b-openshift-service-ca\") pod \"42addc5a-6fe2-4c84-a422-975fd8589f6b\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") "
Sep 11 16:13:56.066008 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.065955    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-maas-api-tls\") pod \"42addc5a-6fe2-4c84-a422-975fd8589f6b\" (UID: \"42addc5a-6fe2-4c84-a422-975fd8589f6b\") "
Sep 11 16:13:56.066289 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.066263    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42addc5a-6fe2-4c84-a422-975fd8589f6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42addc5a-6fe2-4c84-a422-975fd8589f6b" (UID: "42addc5a-6fe2-4c84-a422-975fd8589f6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:13:56.067945 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.067922    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42addc5a-6fe2-4c84-a422-975fd8589f6b" (UID: "42addc5a-6fe2-4c84-a422-975fd8589f6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:13:56.068039 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.067945    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42addc5a-6fe2-4c84-a422-975fd8589f6b-kube-api-access-5x7r4" (OuterVolumeSpecName: "kube-api-access-5x7r4") pod "42addc5a-6fe2-4c84-a422-975fd8589f6b" (UID: "42addc5a-6fe2-4c84-a422-975fd8589f6b"). InnerVolumeSpecName "kube-api-access-5x7r4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:13:56.068114 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.068093    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42addc5a-6fe2-4c84-a422-975fd8589f6b" (UID: "42addc5a-6fe2-4c84-a422-975fd8589f6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:13:56.114553 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.114522    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:56.166989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.166871    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x7r4\" (UniqueName: \"kubernetes.io/projected/42addc5a-6fe2-4c84-a422-975fd8589f6b-kube-api-access-5x7r4\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:13:56.166989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.166899    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42addc5a-6fe2-4c84-a422-975fd8589f6b-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:13:56.166989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.166914    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:13:56.166989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.166927    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42addc5a-6fe2-4c84-a422-975fd8589f6b-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:13:56.167846 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.167825    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:13:56.267731 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.267699    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-plugins-config-volume\") pod \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\" (UID: \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\") "
Sep 11 16:13:56.267922 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.267744    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxqmb\" (UniqueName: \"kubernetes.io/projected/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-kube-api-access-sxqmb\") pod \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\" (UID: \"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e\") "
Sep 11 16:13:56.268138 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.268109    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e" (UID: "6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:13:56.269738 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.269713    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-kube-api-access-sxqmb" (OuterVolumeSpecName: "kube-api-access-sxqmb") pod "6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e" (UID: "6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e"). InnerVolumeSpecName "kube-api-access-sxqmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:13:56.368366 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.368289    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-plugins-config-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:13:56.368366 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.368316    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxqmb\" (UniqueName: \"kubernetes.io/projected/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e-kube-api-access-sxqmb\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:13:56.444088 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.444058    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:56.550385 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.550357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:56.672282 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.672196    2564 generic.go:356] "Generic (PLEG): container finished" podID="6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e" containerID="35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515" exitCode=0
Sep 11 16:13:56.672282 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.672272    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"
Sep 11 16:13:56.672482 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.672276    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs" event={"ID":"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e","Type":"ContainerDied","Data":"35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515"}
Sep 11 16:13:56.672482 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.672332    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs" event={"ID":"6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e","Type":"ContainerDied","Data":"dd36f1de1d92b73e236a7c39518ebf58a164e3f325e6903bbd59e8522fdc1a18"}
Sep 11 16:13:56.672482 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.672348    2564 scope.go:117] "RemoveContainer" containerID="35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515"
Sep 11 16:13:56.673733 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.673709    2564 generic.go:356] "Generic (PLEG): container finished" podID="42addc5a-6fe2-4c84-a422-975fd8589f6b" containerID="1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1" exitCode=0
Sep 11 16:13:56.673842 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.673742    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf" event={"ID":"42addc5a-6fe2-4c84-a422-975fd8589f6b","Type":"ContainerDied","Data":"1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1"}
Sep 11 16:13:56.673842 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.673778    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"
Sep 11 16:13:56.673842 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.673799    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf" event={"ID":"42addc5a-6fe2-4c84-a422-975fd8589f6b","Type":"ContainerDied","Data":"0eac6943955c7c95fffe5ca1f529c658b6e8852af50ef20170b88497fd99da19"}
Sep 11 16:13:56.681151 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.680953    2564 scope.go:117] "RemoveContainer" containerID="35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515"
Sep 11 16:13:56.681314 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:13:56.681285    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515\": container with ID starting with 35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515 not found: ID does not exist" containerID="35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515"
Sep 11 16:13:56.681408 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.681323    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515"} err="failed to get container status \"35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515\": rpc error: code = NotFound desc = could not find container \"35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515\": container with ID starting with 35921d591342acfaa0384966577730a46a31e87cd7a83718846267b9445e9515 not found: ID does not exist"
Sep 11 16:13:56.681408 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.681343    2564 scope.go:117] "RemoveContainer" containerID="1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1"
Sep 11 16:13:56.689088 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.689069    2564 scope.go:117] "RemoveContainer" containerID="1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1"
Sep 11 16:13:56.689363 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:13:56.689342    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1\": container with ID starting with 1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1 not found: ID does not exist" containerID="1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1"
Sep 11 16:13:56.689435 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.689366    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1"} err="failed to get container status \"1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1\": rpc error: code = NotFound desc = could not find container \"1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1\": container with ID starting with 1428489bca54b57e904b460f0677343f7f7ae18e175315a271d71eaf0fb8e3a1 not found: ID does not exist"
Sep 11 16:13:56.697753 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.697732    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"]
Sep 11 16:13:56.704288 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.704269    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-dacdbe-77494fb674-9gdgs"]
Sep 11 16:13:56.715954 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.715935    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"]
Sep 11 16:13:56.719880 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.719860    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-dacdbe-7986fdb8fd-47zjf"]
Sep 11 16:13:56.815578 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.815550    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42addc5a-6fe2-4c84-a422-975fd8589f6b" path="/var/lib/kubelet/pods/42addc5a-6fe2-4c84-a422-975fd8589f6b/volumes"
Sep 11 16:13:56.815886 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:56.815873    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e" path="/var/lib/kubelet/pods/6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e/volumes"
Sep 11 16:13:57.537943 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:57.537909    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:13:58.432142 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:13:58.432112    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:00.635637 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.635590    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b" podUID="53b20f6a-a22b-4dbc-bf62-6c606bd49b35" containerName="main" containerID="cri-o://d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b" gracePeriod=45
Sep 11 16:14:00.891498 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.891446    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:14:00.901646 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.901624    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tls-certs\") pod \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") "
Sep 11 16:14:00.901748 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.901663    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-dshm\") pod \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") "
Sep 11 16:14:00.901748 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.901711    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tmp-dir\") pod \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") "
Sep 11 16:14:00.901748 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.901740    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-model-cache\") pod \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") "
Sep 11 16:14:00.901908 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.901758    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-home\") pod \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") "
Sep 11 16:14:00.901908 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.901788    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k22f\" (UniqueName: \"kubernetes.io/projected/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-kube-api-access-9k22f\") pod \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\" (UID: \"53b20f6a-a22b-4dbc-bf62-6c606bd49b35\") "
Sep 11 16:14:00.902011 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.901964    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53b20f6a-a22b-4dbc-bf62-6c606bd49b35" (UID: "53b20f6a-a22b-4dbc-bf62-6c606bd49b35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:14:00.902072 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.902014    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-model-cache" (OuterVolumeSpecName: "model-cache") pod "53b20f6a-a22b-4dbc-bf62-6c606bd49b35" (UID: "53b20f6a-a22b-4dbc-bf62-6c606bd49b35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:14:00.902072 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.902037    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-home" (OuterVolumeSpecName: "home") pod "53b20f6a-a22b-4dbc-bf62-6c606bd49b35" (UID: "53b20f6a-a22b-4dbc-bf62-6c606bd49b35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:14:00.902184 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.902160    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tmp-dir\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:00.902259 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.902191    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-model-cache\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:00.902259 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.902206    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-home\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:00.903778 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.903757    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53b20f6a-a22b-4dbc-bf62-6c606bd49b35" (UID: "53b20f6a-a22b-4dbc-bf62-6c606bd49b35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:14:00.903860 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.903776    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-dshm" (OuterVolumeSpecName: "dshm") pod "53b20f6a-a22b-4dbc-bf62-6c606bd49b35" (UID: "53b20f6a-a22b-4dbc-bf62-6c606bd49b35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:14:00.903904 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:00.903888    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-kube-api-access-9k22f" (OuterVolumeSpecName: "kube-api-access-9k22f") pod "53b20f6a-a22b-4dbc-bf62-6c606bd49b35" (UID: "53b20f6a-a22b-4dbc-bf62-6c606bd49b35"). InnerVolumeSpecName "kube-api-access-9k22f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:14:01.003154 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.003126    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k22f\" (UniqueName: \"kubernetes.io/projected/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-kube-api-access-9k22f\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:01.003154 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.003151    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-tls-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:01.003337 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.003161    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53b20f6a-a22b-4dbc-bf62-6c606bd49b35-dshm\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:01.523365 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.523336    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:01.698181 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.698156    2564 generic.go:356] "Generic (PLEG): container finished" podID="53b20f6a-a22b-4dbc-bf62-6c606bd49b35" containerID="d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b" exitCode=0
Sep 11 16:14:01.698572 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.698195    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"
Sep 11 16:14:01.698572 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.698223    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b" event={"ID":"53b20f6a-a22b-4dbc-bf62-6c606bd49b35","Type":"ContainerDied","Data":"d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b"}
Sep 11 16:14:01.698572 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.698274    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b" event={"ID":"53b20f6a-a22b-4dbc-bf62-6c606bd49b35","Type":"ContainerDied","Data":"caa0b4aaa3fb440d13d9bde313080bd21574aee5fe5be1e9262ab10e13ac4d89"}
Sep 11 16:14:01.698572 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.698291    2564 scope.go:117] "RemoveContainer" containerID="d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b"
Sep 11 16:14:01.710429 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.710404    2564 scope.go:117] "RemoveContainer" containerID="d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b"
Sep 11 16:14:01.710787 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:14:01.710766    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b\": container with ID starting with d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b not found: ID does not exist" containerID="d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b"
Sep 11 16:14:01.710882 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.710794    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b"} err="failed to get container status \"d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b\": rpc error: code = NotFound desc = could not find container \"d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b\": container with ID starting with d4825211db874bc2e6e4dedb2f23b95419eacfe660186e04fda4c4504f0d605b not found: ID does not exist"
Sep 11 16:14:01.725730 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.725703    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"]
Sep 11 16:14:01.732069 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.732050    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-dacdbe/explicit-model-dacdbe-kserve-d94f77d7f-7nl2b"]
Sep 11 16:14:01.982024 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:01.982001    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:02.081899 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:02.081858    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:02.220723 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:02.220698    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:02.691142 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:02.691107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:02.810099 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:02.810066    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53b20f6a-a22b-4dbc-bf62-6c606bd49b35" path="/var/lib/kubelet/pods/53b20f6a-a22b-4dbc-bf62-6c606bd49b35/volumes"
Sep 11 16:14:02.839341 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:02.839310    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:04.027488 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:04.027455    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:04.126969 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:04.126940    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:04.280703 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:04.280611    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:04.756984 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:04.756952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:05.097859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:05.097817    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:05.478297 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:05.478204    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:05.964578 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:05.964535    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:06.108612 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:06.108577    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:06.357203 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:06.357165    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:06.479690 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:06.479657    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:06.581454 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:06.581425    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:07.407006 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:07.406970    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:08.223502 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.223468    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"]
Sep 11 16:14:08.224015 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.224000    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e" containerName="payload-pre-processing"
Sep 11 16:14:08.224056 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.224019    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e" containerName="payload-pre-processing"
Sep 11 16:14:08.224056 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.224034    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53b20f6a-a22b-4dbc-bf62-6c606bd49b35" containerName="main"
Sep 11 16:14:08.224056 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.224043    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="53b20f6a-a22b-4dbc-bf62-6c606bd49b35" containerName="main"
Sep 11 16:14:08.224153 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.224066    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42addc5a-6fe2-4c84-a422-975fd8589f6b" containerName="maas-api"
Sep 11 16:14:08.224153 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.224075    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="42addc5a-6fe2-4c84-a422-975fd8589f6b" containerName="maas-api"
Sep 11 16:14:08.224153 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.224146    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7ef2ef-f84b-473c-91a9-4e6d5ecdce8e" containerName="payload-pre-processing"
Sep 11 16:14:08.224262 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.224158    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="42addc5a-6fe2-4c84-a422-975fd8589f6b" containerName="maas-api"
Sep 11 16:14:08.224262 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.224172    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="53b20f6a-a22b-4dbc-bf62-6c606bd49b35" containerName="main"
Sep 11 16:14:08.228634 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.228604    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.232026 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.232004    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-faa154ea-serving-cert\""
Sep 11 16:14:08.242041 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.242016    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"]
Sep 11 16:14:08.264032 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.264007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-metrics-certs\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.264137 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.264039    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-maas-api-tls\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.264137 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.264101    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbbvm\" (UniqueName: \"kubernetes.io/projected/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-kube-api-access-fbbvm\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.264209 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.264163    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-openshift-service-ca\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.279531 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.279504    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:08.365060 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.365038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbvm\" (UniqueName: \"kubernetes.io/projected/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-kube-api-access-fbbvm\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.365177 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.365088    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-openshift-service-ca\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.365177 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.365155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-metrics-certs\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.365340 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.365186    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-maas-api-tls\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.365899 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.365873    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-openshift-service-ca\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.367528 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.367507    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-metrics-certs\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.367594 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.367556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-maas-api-tls\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.380544 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.380521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbvm\" (UniqueName: \"kubernetes.io/projected/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-kube-api-access-fbbvm\") pod \"maas-api-e2e-mt-faa154ea-7c958d6588-59pvm\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.392564 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.392537    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:08.493315 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.493230    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:08.539804 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.539778    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:08.814732 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.814709    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"]
Sep 11 16:14:08.816487 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:14:08.816454    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad45e8fa_492c_45b0_ba5b_9c47f58e5387.slice/crio-d4c140051d72800521a1ce4c4d77f2c2ef9632d08952c40937cdcb2fb49a8e8b WatchSource:0}: Error finding container d4c140051d72800521a1ce4c4d77f2c2ef9632d08952c40937cdcb2fb49a8e8b: Status 404 returned error can't find the container with id d4c140051d72800521a1ce4c4d77f2c2ef9632d08952c40937cdcb2fb49a8e8b
Sep 11 16:14:08.946923 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:08.946892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:09.069276 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.069196    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:09.247878 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.247847    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"]
Sep 11 16:14:09.252281 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.252255    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:09.255744 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.255608    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-faa154ea-dockercfg-49qrj\""
Sep 11 16:14:09.255744 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.255735    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-faa154ea\""
Sep 11 16:14:09.273107 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.273083    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-plugins-config-volume\") pod \"payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj\" (UID: \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\") " pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:09.273223 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.273119    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfvgg\" (UniqueName: \"kubernetes.io/projected/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-kube-api-access-rfvgg\") pod \"payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj\" (UID: \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\") " pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:09.273223 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.273208    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"]
Sep 11 16:14:09.373694 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.373664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-plugins-config-volume\") pod \"payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj\" (UID: \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\") " pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:09.373790 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.373707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfvgg\" (UniqueName: \"kubernetes.io/projected/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-kube-api-access-rfvgg\") pod \"payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj\" (UID: \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\") " pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:09.374397 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.374377    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-plugins-config-volume\") pod \"payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj\" (UID: \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\") " pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:09.382668 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.382642    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfvgg\" (UniqueName: \"kubernetes.io/projected/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-kube-api-access-rfvgg\") pod \"payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj\" (UID: \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\") " pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:09.564487 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.564451    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:09.734981 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.734947    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm" event={"ID":"ad45e8fa-492c-45b0-ba5b-9c47f58e5387","Type":"ContainerStarted","Data":"e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960"}
Sep 11 16:14:09.735149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.735005    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm" event={"ID":"ad45e8fa-492c-45b0-ba5b-9c47f58e5387","Type":"ContainerStarted","Data":"d4c140051d72800521a1ce4c4d77f2c2ef9632d08952c40937cdcb2fb49a8e8b"}
Sep 11 16:14:09.735149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.735082    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:09.757565 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.757487    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm" podStartSLOduration=1.244897101 podStartE2EDuration="1.75747271s" totalImagesPullingTime="512.575609ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:14:09.755976804 +0000 UTC m=+1143.473951943" watchObservedRunningTime="2026-09-11 16:14:09.75747271 +0000 UTC m=+1143.475447830"
Sep 11 16:14:09.865744 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:09.865723    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"]
Sep 11 16:14:09.867305 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:14:09.867277    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb6d84b_af34_4b0c_b191_14df3a7b56b8.slice/crio-0429e6142b01ff1d95b6f53d0c4c68e641e5c56db8875ca6199b6c8c0e13ed36 WatchSource:0}: Error finding container 0429e6142b01ff1d95b6f53d0c4c68e641e5c56db8875ca6199b6c8c0e13ed36: Status 404 returned error can't find the container with id 0429e6142b01ff1d95b6f53d0c4c68e641e5c56db8875ca6199b6c8c0e13ed36
Sep 11 16:14:10.741574 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:10.741544    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj" event={"ID":"1eb6d84b-af34-4b0c-b191-14df3a7b56b8","Type":"ContainerStarted","Data":"8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40"}
Sep 11 16:14:10.741968 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:10.741603    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj" event={"ID":"1eb6d84b-af34-4b0c-b191-14df3a7b56b8","Type":"ContainerStarted","Data":"0429e6142b01ff1d95b6f53d0c4c68e641e5c56db8875ca6199b6c8c0e13ed36"}
Sep 11 16:14:10.741968 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:10.741835    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:10.766933 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:10.766881    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj" podStartSLOduration=1.7668635240000001 podStartE2EDuration="1.766863524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:14:10.765022855 +0000 UTC m=+1144.482997994" watchObservedRunningTime="2026-09-11 16:14:10.766863524 +0000 UTC m=+1144.484838644"
Sep 11 16:14:15.747220 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:15.747186    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:21.747195 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:21.747164    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:30.880358 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:30.880272    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:30.986372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:30.986338    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:31.582161 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:31.582127    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:31.706081 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:31.706050    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:32.000270 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:32.000155    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:32.104292 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:32.104259    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:33.081714 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:33.081681    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:33.860149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:33.860108    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:34.097578 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:34.097541    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:36.061662 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:36.061624    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:38.390236 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:38.390205    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:38.491772 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:38.491744    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:39.861099 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:39.861065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:40.755369 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:40.755329    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:41.611433 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:41.611402    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:42.603652 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:42.603613    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:46.898638 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:46.898584    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:49.357033 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.356990    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"]
Sep 11 16:14:49.357497 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.357211    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm" podUID="ad45e8fa-492c-45b0-ba5b-9c47f58e5387" containerName="maas-api" containerID="cri-o://e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960" gracePeriod=30
Sep 11 16:14:49.399462 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.399435    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:49.428294 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.428263    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"]
Sep 11 16:14:49.428485 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.428462    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj" podUID="1eb6d84b-af34-4b0c-b191-14df3a7b56b8" containerName="payload-processing" containerID="cri-o://8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40" gracePeriod=30
Sep 11 16:14:49.498595 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.498550    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:49.598476 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.598405    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:49.674121 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.674098    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:49.676327 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.676296    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:49.701518 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.701020    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:49.780644 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.780605    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-metrics-certs\") pod \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") "
Sep 11 16:14:49.780834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.780652    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-maas-api-tls\") pod \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") "
Sep 11 16:14:49.780834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.780698    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfvgg\" (UniqueName: \"kubernetes.io/projected/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-kube-api-access-rfvgg\") pod \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\" (UID: \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\") "
Sep 11 16:14:49.780834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.780741    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fbbvm\" (UniqueName: \"kubernetes.io/projected/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-kube-api-access-fbbvm\") pod \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") "
Sep 11 16:14:49.780834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.780765    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-openshift-service-ca\") pod \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\" (UID: \"ad45e8fa-492c-45b0-ba5b-9c47f58e5387\") "
Sep 11 16:14:49.780834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.780826    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-plugins-config-volume\") pod \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\" (UID: \"1eb6d84b-af34-4b0c-b191-14df3a7b56b8\") "
Sep 11 16:14:49.781169 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.781136    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad45e8fa-492c-45b0-ba5b-9c47f58e5387" (UID: "ad45e8fa-492c-45b0-ba5b-9c47f58e5387"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:14:49.781429 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.781402    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1eb6d84b-af34-4b0c-b191-14df3a7b56b8" (UID: "1eb6d84b-af34-4b0c-b191-14df3a7b56b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:14:49.782799 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.782768    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad45e8fa-492c-45b0-ba5b-9c47f58e5387" (UID: "ad45e8fa-492c-45b0-ba5b-9c47f58e5387"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:14:49.782960 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.782936    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad45e8fa-492c-45b0-ba5b-9c47f58e5387" (UID: "ad45e8fa-492c-45b0-ba5b-9c47f58e5387"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:14:49.783150 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.783127    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-kube-api-access-rfvgg" (OuterVolumeSpecName: "kube-api-access-rfvgg") pod "1eb6d84b-af34-4b0c-b191-14df3a7b56b8" (UID: "1eb6d84b-af34-4b0c-b191-14df3a7b56b8"). InnerVolumeSpecName "kube-api-access-rfvgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:14:49.783234 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.783162    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-kube-api-access-fbbvm" (OuterVolumeSpecName: "kube-api-access-fbbvm") pod "ad45e8fa-492c-45b0-ba5b-9c47f58e5387" (UID: "ad45e8fa-492c-45b0-ba5b-9c47f58e5387"). InnerVolumeSpecName "kube-api-access-fbbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:14:49.882420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.882350    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:49.882420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.882374    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:49.882420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.882384    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfvgg\" (UniqueName: \"kubernetes.io/projected/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-kube-api-access-rfvgg\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:49.882420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.882393    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbbvm\" (UniqueName: \"kubernetes.io/projected/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-kube-api-access-fbbvm\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:49.882420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.882403    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad45e8fa-492c-45b0-ba5b-9c47f58e5387-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:49.882420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.882412    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1eb6d84b-af34-4b0c-b191-14df3a7b56b8-plugins-config-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:14:49.924914 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.924881    2564 generic.go:356] "Generic (PLEG): container finished" podID="ad45e8fa-492c-45b0-ba5b-9c47f58e5387" containerID="e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960" exitCode=0
Sep 11 16:14:49.925052 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.924937    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"
Sep 11 16:14:49.925052 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.924927    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm" event={"ID":"ad45e8fa-492c-45b0-ba5b-9c47f58e5387","Type":"ContainerDied","Data":"e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960"}
Sep 11 16:14:49.925169 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.925065    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm" event={"ID":"ad45e8fa-492c-45b0-ba5b-9c47f58e5387","Type":"ContainerDied","Data":"d4c140051d72800521a1ce4c4d77f2c2ef9632d08952c40937cdcb2fb49a8e8b"}
Sep 11 16:14:49.925169 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.925099    2564 scope.go:117] "RemoveContainer" containerID="e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960"
Sep 11 16:14:49.926398 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.926380    2564 generic.go:356] "Generic (PLEG): container finished" podID="1eb6d84b-af34-4b0c-b191-14df3a7b56b8" containerID="8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40" exitCode=0
Sep 11 16:14:49.926506 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.926442    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"
Sep 11 16:14:49.926506 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.926450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj" event={"ID":"1eb6d84b-af34-4b0c-b191-14df3a7b56b8","Type":"ContainerDied","Data":"8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40"}
Sep 11 16:14:49.926506 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.926480    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj" event={"ID":"1eb6d84b-af34-4b0c-b191-14df3a7b56b8","Type":"ContainerDied","Data":"0429e6142b01ff1d95b6f53d0c4c68e641e5c56db8875ca6199b6c8c0e13ed36"}
Sep 11 16:14:49.935330 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.935314    2564 scope.go:117] "RemoveContainer" containerID="e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960"
Sep 11 16:14:49.935608 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:14:49.935591    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960\": container with ID starting with e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960 not found: ID does not exist" containerID="e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960"
Sep 11 16:14:49.935657 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.935614    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960"} err="failed to get container status \"e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960\": rpc error: code = NotFound desc = could not find container \"e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960\": container with ID starting with e5716bc83441510e085ce66c9db3a3cc1a96a67bf104c0a2c981b0463285a960 not found: ID does not exist"
Sep 11 16:14:49.935657 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.935629    2564 scope.go:117] "RemoveContainer" containerID="8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40"
Sep 11 16:14:49.942781 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.942767    2564 scope.go:117] "RemoveContainer" containerID="8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40"
Sep 11 16:14:49.943035 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:14:49.943017    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40\": container with ID starting with 8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40 not found: ID does not exist" containerID="8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40"
Sep 11 16:14:49.943100 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.943040    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40"} err="failed to get container status \"8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40\": rpc error: code = NotFound desc = could not find container \"8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40\": container with ID starting with 8f7781afcd2853e9679fc6812a2060e8fa25d3530d2b6687384980dd6802df40 not found: ID does not exist"
Sep 11 16:14:49.949061 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.949042    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"]
Sep 11 16:14:49.955339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.955318    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-faa154ea-7c958d6588-59pvm"]
Sep 11 16:14:49.970642 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.970619    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"]
Sep 11 16:14:49.974787 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:49.974766    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-faa154ea-548d8d5c65-rbxrj"]
Sep 11 16:14:50.039532 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:50.039482    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:50.137121 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:50.137066    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:50.810806 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:50.810776    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb6d84b-af34-4b0c-b191-14df3a7b56b8" path="/var/lib/kubelet/pods/1eb6d84b-af34-4b0c-b191-14df3a7b56b8/volumes"
Sep 11 16:14:50.811133 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:50.811126    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad45e8fa-492c-45b0-ba5b-9c47f58e5387" path="/var/lib/kubelet/pods/ad45e8fa-492c-45b0-ba5b-9c47f58e5387/volumes"
Sep 11 16:14:51.163748 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:51.163672    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:51.963827 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:51.963797    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:56.700583 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:56.700552    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:56.838556 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:56.838522    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:56.965856 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:56.965780    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:57.129718 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:57.129687    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:57.245219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:57.245142    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:57.462887 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:57.462859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:57.894369 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:57.894340    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:57.991738 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:57.991688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:58.177780 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:58.177706    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:58.502476 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:58.502403    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:58.625780 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:58.625749    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:58.784275 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:58.784178    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:14:59.585617 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:14:59.585579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:02.269795 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:02.269761    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:05.464566 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:05.464529    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:05.570922 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:05.570893    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:05.672590 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:05.672559    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:05.823850 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:05.823816    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:06.691265 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:06.691202    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"]
Sep 11 16:15:06.691709 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:06.691548    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t" podUID="34bc2cca-1857-4e31-ae3a-659a4af87db7" containerName="maas-api" containerID="cri-o://1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6" gracePeriod=30
Sep 11 16:15:06.718857 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:06.718827    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:06.818100 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:06.818064    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:06.917298 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:06.917270    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:06.936390 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:06.936367    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:15:07.004165 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.004088    2564 generic.go:356] "Generic (PLEG): container finished" podID="34bc2cca-1857-4e31-ae3a-659a4af87db7" containerID="1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6" exitCode=0
Sep 11 16:15:07.004344 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.004174    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"
Sep 11 16:15:07.004344 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.004217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t" event={"ID":"34bc2cca-1857-4e31-ae3a-659a4af87db7","Type":"ContainerDied","Data":"1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6"}
Sep 11 16:15:07.004344 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.004328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t" event={"ID":"34bc2cca-1857-4e31-ae3a-659a4af87db7","Type":"ContainerDied","Data":"02752451b5432feee474cda6475f411f02bd7d2d656a679d35d19f1246782e95"}
Sep 11 16:15:07.004505 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.004365    2564 scope.go:117] "RemoveContainer" containerID="1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6"
Sep 11 16:15:07.012533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.012518    2564 scope.go:117] "RemoveContainer" containerID="1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6"
Sep 11 16:15:07.012780 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:15:07.012763    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6\": container with ID starting with 1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6 not found: ID does not exist" containerID="1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6"
Sep 11 16:15:07.012826 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.012786    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6"} err="failed to get container status \"1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6\": rpc error: code = NotFound desc = could not find container \"1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6\": container with ID starting with 1096b849931bfcf2328a167afc1a2b1f237b0ab21a4c64c384c239a2669636c6 not found: ID does not exist"
Sep 11 16:15:07.017082 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.017062    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bc2cca-1857-4e31-ae3a-659a4af87db7-openshift-service-ca\") pod \"34bc2cca-1857-4e31-ae3a-659a4af87db7\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") "
Sep 11 16:15:07.017179 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.017130    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-maas-api-tls\") pod \"34bc2cca-1857-4e31-ae3a-659a4af87db7\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") "
Sep 11 16:15:07.017232 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.017205    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhtrd\" (UniqueName: \"kubernetes.io/projected/34bc2cca-1857-4e31-ae3a-659a4af87db7-kube-api-access-vhtrd\") pod \"34bc2cca-1857-4e31-ae3a-659a4af87db7\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") "
Sep 11 16:15:07.017297 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.017282    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-metrics-certs\") pod \"34bc2cca-1857-4e31-ae3a-659a4af87db7\" (UID: \"34bc2cca-1857-4e31-ae3a-659a4af87db7\") "
Sep 11 16:15:07.017416 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.017391    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34bc2cca-1857-4e31-ae3a-659a4af87db7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34bc2cca-1857-4e31-ae3a-659a4af87db7" (UID: "34bc2cca-1857-4e31-ae3a-659a4af87db7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:15:07.017711 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.017688    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bc2cca-1857-4e31-ae3a-659a4af87db7-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:15:07.019317 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.019294    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:07.019595 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.019571    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34bc2cca-1857-4e31-ae3a-659a4af87db7" (UID: "34bc2cca-1857-4e31-ae3a-659a4af87db7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:15:07.019698 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.019676    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "34bc2cca-1857-4e31-ae3a-659a4af87db7" (UID: "34bc2cca-1857-4e31-ae3a-659a4af87db7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:15:07.019792 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.019771    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bc2cca-1857-4e31-ae3a-659a4af87db7-kube-api-access-vhtrd" (OuterVolumeSpecName: "kube-api-access-vhtrd") pod "34bc2cca-1857-4e31-ae3a-659a4af87db7" (UID: "34bc2cca-1857-4e31-ae3a-659a4af87db7"). InnerVolumeSpecName "kube-api-access-vhtrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:15:07.096112 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.096084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:07.118929 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.118897    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:15:07.118929 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.118923    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhtrd\" (UniqueName: \"kubernetes.io/projected/34bc2cca-1857-4e31-ae3a-659a4af87db7-kube-api-access-vhtrd\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:15:07.118929 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.118932    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34bc2cca-1857-4e31-ae3a-659a4af87db7-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:15:07.326796 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.326767    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"]
Sep 11 16:15:07.335436 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.335411    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8e3121-5c95686d6-l4f2t"]
Sep 11 16:15:07.365715 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.365692    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:07.467325 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:07.467304    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:08.227217 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:08.227185    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:08.810484 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:08.810453    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34bc2cca-1857-4e31-ae3a-659a4af87db7" path="/var/lib/kubelet/pods/34bc2cca-1857-4e31-ae3a-659a4af87db7/volumes"
Sep 11 16:15:08.898911 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:08.898873    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:12.750036 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:12.749986    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:15.124361 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:15.124328    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:15.365800 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:15.365766    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:16.353225 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:16.353188    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:17.463738 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:17.463705    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:17.568639 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:17.568599    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:18.471975 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:18.471946    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:19.384829 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:19.384778    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:20.162738 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.162701    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"]
Sep 11 16:15:20.163376 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.163342    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34bc2cca-1857-4e31-ae3a-659a4af87db7" containerName="maas-api"
Sep 11 16:15:20.163376 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.163369    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="34bc2cca-1857-4e31-ae3a-659a4af87db7" containerName="maas-api"
Sep 11 16:15:20.163550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.163383    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb6d84b-af34-4b0c-b191-14df3a7b56b8" containerName="payload-processing"
Sep 11 16:15:20.163550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.163392    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb6d84b-af34-4b0c-b191-14df3a7b56b8" containerName="payload-processing"
Sep 11 16:15:20.163550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.163406    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad45e8fa-492c-45b0-ba5b-9c47f58e5387" containerName="maas-api"
Sep 11 16:15:20.163550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.163414    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad45e8fa-492c-45b0-ba5b-9c47f58e5387" containerName="maas-api"
Sep 11 16:15:20.163550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.163520    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad45e8fa-492c-45b0-ba5b-9c47f58e5387" containerName="maas-api"
Sep 11 16:15:20.163550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.163537    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="34bc2cca-1857-4e31-ae3a-659a4af87db7" containerName="maas-api"
Sep 11 16:15:20.163550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.163546    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb6d84b-af34-4b0c-b191-14df3a7b56b8" containerName="payload-processing"
Sep 11 16:15:20.168097 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.168076    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.171008 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.170989    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f7c21d-serving-cert\""
Sep 11 16:15:20.180420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.180399    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"]
Sep 11 16:15:20.217764 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.217739    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.217868 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.217800    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.217868 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.217820    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwx2\" (UniqueName: \"kubernetes.io/projected/4affd20a-c6ef-4ffe-825a-c414e9426c45-kube-api-access-7lwx2\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.217868 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.217857    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4affd20a-c6ef-4ffe-825a-c414e9426c45-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.318649 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.318619    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.318649 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.318649    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwx2\" (UniqueName: \"kubernetes.io/projected/4affd20a-c6ef-4ffe-825a-c414e9426c45-kube-api-access-7lwx2\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.318850 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.318681    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4affd20a-c6ef-4ffe-825a-c414e9426c45-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.318850 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.318706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.319329 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.319307    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4affd20a-c6ef-4ffe-825a-c414e9426c45-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.321076 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.321045    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.321156 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.321045    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.328428 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.328409    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwx2\" (UniqueName: \"kubernetes.io/projected/4affd20a-c6ef-4ffe-825a-c414e9426c45-kube-api-access-7lwx2\") pod \"maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.337572 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.337541    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:20.439869 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.439808    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:20.480714 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.480695    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:20.752341 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.752255    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"]
Sep 11 16:15:20.755286 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:15:20.755223    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4affd20a_c6ef_4ffe_825a_c414e9426c45.slice/crio-df366535bdfe2d4e3b7ad23e693ec0634851a8cd2c4ea336d52af0d90d4f1ebf WatchSource:0}: Error finding container df366535bdfe2d4e3b7ad23e693ec0634851a8cd2c4ea336d52af0d90d4f1ebf: Status 404 returned error can't find the container with id df366535bdfe2d4e3b7ad23e693ec0634851a8cd2c4ea336d52af0d90d4f1ebf
Sep 11 16:15:20.774674 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.774645    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:20.865916 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:20.865886    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:21.069906 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:21.069876    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd" event={"ID":"4affd20a-c6ef-4ffe-825a-c414e9426c45","Type":"ContainerStarted","Data":"df366535bdfe2d4e3b7ad23e693ec0634851a8cd2c4ea336d52af0d90d4f1ebf"}
Sep 11 16:15:22.075235 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:22.075201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd" event={"ID":"4affd20a-c6ef-4ffe-825a-c414e9426c45","Type":"ContainerStarted","Data":"b7eb958772954c35d92d1b0d54e799d497665243881f32e495cc7be721d94b81"}
Sep 11 16:15:22.075609 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:22.075392    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:22.099940 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:22.099898    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd" podStartSLOduration=1.620938327 podStartE2EDuration="2.099884794s" totalImagesPullingTime="478.946467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:22.09826788 +0000 UTC m=+1215.816243015" watchObservedRunningTime="2026-09-11 16:15:22.099884794 +0000 UTC m=+1215.817859914"
Sep 11 16:15:24.488687 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:24.488655    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:25.281292 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.281259    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"]
Sep 11 16:15:25.281531 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.281501    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c" podUID="97e7e7d0-1129-4dfd-8efa-74617a8dc086" containerName="maas-api" containerID="cri-o://5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1" gracePeriod=30
Sep 11 16:15:25.325815 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.325787    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:25.420610 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.420579    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:25.501875 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.501843    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:25.529035 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.529015    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:15:25.559554 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.559477    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e7e7d0-1129-4dfd-8efa-74617a8dc086-openshift-service-ca\") pod \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") "
Sep 11 16:15:25.559554 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.559545    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-maas-api-tls\") pod \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") "
Sep 11 16:15:25.559720 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.559571    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnggp\" (UniqueName: \"kubernetes.io/projected/97e7e7d0-1129-4dfd-8efa-74617a8dc086-kube-api-access-rnggp\") pod \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") "
Sep 11 16:15:25.559720 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.559641    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-metrics-certs\") pod \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\" (UID: \"97e7e7d0-1129-4dfd-8efa-74617a8dc086\") "
Sep 11 16:15:25.560570 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.560529    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e7e7d0-1129-4dfd-8efa-74617a8dc086-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "97e7e7d0-1129-4dfd-8efa-74617a8dc086" (UID: "97e7e7d0-1129-4dfd-8efa-74617a8dc086"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:15:25.562333 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.562294    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e7e7d0-1129-4dfd-8efa-74617a8dc086-kube-api-access-rnggp" (OuterVolumeSpecName: "kube-api-access-rnggp") pod "97e7e7d0-1129-4dfd-8efa-74617a8dc086" (UID: "97e7e7d0-1129-4dfd-8efa-74617a8dc086"). InnerVolumeSpecName "kube-api-access-rnggp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:15:25.562470 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.562385    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "97e7e7d0-1129-4dfd-8efa-74617a8dc086" (UID: "97e7e7d0-1129-4dfd-8efa-74617a8dc086"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:15:25.563166 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.562844    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "97e7e7d0-1129-4dfd-8efa-74617a8dc086" (UID: "97e7e7d0-1129-4dfd-8efa-74617a8dc086"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:15:25.588209 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.588185    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:25.660892 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.660868    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e7e7d0-1129-4dfd-8efa-74617a8dc086-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:15:25.660892 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.660893    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:15:25.661019 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.660902    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnggp\" (UniqueName: \"kubernetes.io/projected/97e7e7d0-1129-4dfd-8efa-74617a8dc086-kube-api-access-rnggp\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:15:25.661019 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.660910    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97e7e7d0-1129-4dfd-8efa-74617a8dc086-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:15:25.981188 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:25.981155    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:26.093803 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.093772    2564 generic.go:356] "Generic (PLEG): container finished" podID="97e7e7d0-1129-4dfd-8efa-74617a8dc086" containerID="5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1" exitCode=0
Sep 11 16:15:26.094021 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.093810    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"
Sep 11 16:15:26.094021 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.093856    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c" event={"ID":"97e7e7d0-1129-4dfd-8efa-74617a8dc086","Type":"ContainerDied","Data":"5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1"}
Sep 11 16:15:26.094021 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.093912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c" event={"ID":"97e7e7d0-1129-4dfd-8efa-74617a8dc086","Type":"ContainerDied","Data":"7e10fa7fa07f9c92de871f97e935b6a9e6c1dda0132adb6b4031d5ace50800bb"}
Sep 11 16:15:26.094021 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.093937    2564 scope.go:117] "RemoveContainer" containerID="5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1"
Sep 11 16:15:26.102711 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.102693    2564 scope.go:117] "RemoveContainer" containerID="5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1"
Sep 11 16:15:26.103000 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:15:26.102983    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1\": container with ID starting with 5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1 not found: ID does not exist" containerID="5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1"
Sep 11 16:15:26.103064 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.103009    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1"} err="failed to get container status \"5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1\": rpc error: code = NotFound desc = could not find container \"5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1\": container with ID starting with 5e7c38d39650d2e5abbf43c62f694b80a59533e55e9185bc98527e73d6addab1 not found: ID does not exist"
Sep 11 16:15:26.104443 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.104421    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:26.115092 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.115069    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"]
Sep 11 16:15:26.126564 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.126545    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2b1bd5-76fd6d7df-t6q6c"]
Sep 11 16:15:26.810653 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:26.810623    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97e7e7d0-1129-4dfd-8efa-74617a8dc086" path="/var/lib/kubelet/pods/97e7e7d0-1129-4dfd-8efa-74617a8dc086/volumes"
Sep 11 16:15:27.068180 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:27.068096    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:28.084966 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:28.084938    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:15:32.504551 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:32.504518    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:32.619613 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:32.619582    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:32.863044 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:32.863020    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:33.351913 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:33.351882    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:33.717267 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:33.717174    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:34.288326 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:34.288295    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:36.204900 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:36.204863    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:36.856502 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:36.856469    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:36.986227 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:36.986193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:37.569299 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:37.569267    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:38.800311 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.800272    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"]
Sep 11 16:15:38.800814 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.800792    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97e7e7d0-1129-4dfd-8efa-74617a8dc086" containerName="maas-api"
Sep 11 16:15:38.800897 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.800817    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="97e7e7d0-1129-4dfd-8efa-74617a8dc086" containerName="maas-api"
Sep 11 16:15:38.800951 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.800898    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="97e7e7d0-1129-4dfd-8efa-74617a8dc086" containerName="maas-api"
Sep 11 16:15:38.803641 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.803620    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.819323 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.819299    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-59f108-serving-cert\""
Sep 11 16:15:38.866224 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.866196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85eb56c-fc76-4349-b115-1c5f50046a7c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.866381 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.866306    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6g8t\" (UniqueName: \"kubernetes.io/projected/c85eb56c-fc76-4349-b115-1c5f50046a7c-kube-api-access-h6g8t\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.866381 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.866358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.866505 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.866403    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.900706 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.900673    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"]
Sep 11 16:15:38.967759 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.967731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.967887 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.967787    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85eb56c-fc76-4349-b115-1c5f50046a7c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.967887 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.967851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6g8t\" (UniqueName: \"kubernetes.io/projected/c85eb56c-fc76-4349-b115-1c5f50046a7c-kube-api-access-h6g8t\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.968017 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.967889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.968068 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:15:38.968016    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-serving-cert: secret "maas-api-e2e-shared-a-w3-59f108-serving-cert" not found
Sep 11 16:15:38.968145 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:15:38.968128    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-maas-api-tls podName:c85eb56c-fc76-4349-b115-1c5f50046a7c nodeName:}" failed. No retries permitted until 2026-09-11 16:15:39.46810559 +0000 UTC m=+1233.186080690 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-maas-api-tls") pod "maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9" (UID: "c85eb56c-fc76-4349-b115-1c5f50046a7c") : secret "maas-api-e2e-shared-a-w3-59f108-serving-cert" not found
Sep 11 16:15:38.968639 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.968617    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85eb56c-fc76-4349-b115-1c5f50046a7c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.970830 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.970806    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:38.977573 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.977550    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:38.994520 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:38.994497    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6g8t\" (UniqueName: \"kubernetes.io/projected/c85eb56c-fc76-4349-b115-1c5f50046a7c-kube-api-access-h6g8t\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:39.135985 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:39.135963    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:39.473144 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:39.473077    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:39.475961 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:39.475934    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:39.714190 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:39.714149    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:39.788623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:39.788587    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:39.968547 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:39.968515    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:40.000731 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:40.000629    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"]
Sep 11 16:15:40.003164 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:15:40.003139    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85eb56c_fc76_4349_b115_1c5f50046a7c.slice/crio-3b55cd189b8a0f3f775803853d93051e1017150daf7eff85eb208f54de9e5f61 WatchSource:0}: Error finding container 3b55cd189b8a0f3f775803853d93051e1017150daf7eff85eb208f54de9e5f61: Status 404 returned error can't find the container with id 3b55cd189b8a0f3f775803853d93051e1017150daf7eff85eb208f54de9e5f61
Sep 11 16:15:40.154298 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:40.154269    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9" event={"ID":"c85eb56c-fc76-4349-b115-1c5f50046a7c","Type":"ContainerStarted","Data":"3b55cd189b8a0f3f775803853d93051e1017150daf7eff85eb208f54de9e5f61"}
Sep 11 16:15:40.569029 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:40.568983    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:41.161422 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:41.161390    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9" event={"ID":"c85eb56c-fc76-4349-b115-1c5f50046a7c","Type":"ContainerStarted","Data":"b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395"}
Sep 11 16:15:41.161834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:41.161527    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:41.204916 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:41.204862    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9" podStartSLOduration=2.779279795 podStartE2EDuration="3.204850167s" totalImagesPullingTime="425.570372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:41.204149261 +0000 UTC m=+1234.922124401" watchObservedRunningTime="2026-09-11 16:15:41.204850167 +0000 UTC m=+1234.922825286"
Sep 11 16:15:42.003775 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:42.003742    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:42.512588 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:42.512556    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:42.761902 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:42.761849    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:43.361468 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:43.361440    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:43.485932 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:43.485896    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:44.319541 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:44.319506    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:44.627565 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:44.627490    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:46.247281 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:46.247216    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:46.916990 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:46.916962    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:47.177728 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:47.177651    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:15:47.196668 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:47.196643    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:47.995143 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:47.995110    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:49.187943 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:49.187908    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:50.380770 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.380741    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"]
Sep 11 16:15:50.384135 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.384119    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.390935 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:15:50.390893    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-e2e-shared-b-w2-320b99-serving-cert\" is forbidden: User \"system:node:ip-10-0-141-101.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-101.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-320b99-serving-cert\"" type="*v1.Secret"
Sep 11 16:15:50.459100 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.459070    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.459204 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.459107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm5tq\" (UniqueName: \"kubernetes.io/projected/3aff9c12-7bba-41b8-a7ec-839675a54458-kube-api-access-nm5tq\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.459204 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.459134    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.459204 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.459152    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff9c12-7bba-41b8-a7ec-839675a54458-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.475868 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.475842    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"]
Sep 11 16:15:50.559832 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.559799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.559993 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.559838    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm5tq\" (UniqueName: \"kubernetes.io/projected/3aff9c12-7bba-41b8-a7ec-839675a54458-kube-api-access-nm5tq\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.559993 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.559867    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.559993 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.559889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff9c12-7bba-41b8-a7ec-839675a54458-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.560661 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.560638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff9c12-7bba-41b8-a7ec-839675a54458-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.562176 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.562159    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:50.589079 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:50.589051    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm5tq\" (UniqueName: \"kubernetes.io/projected/3aff9c12-7bba-41b8-a7ec-839675a54458-kube-api-access-nm5tq\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:51.393953 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:51.393924    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-320b99-serving-cert\""
Sep 11 16:15:51.403093 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:51.403069    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:51.423016 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:51.422990    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:51.594184 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:51.594158    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:51.709326 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:51.709292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:51.889761 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:51.889737    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"]
Sep 11 16:15:51.891341 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:15:51.891317    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aff9c12_7bba_41b8_a7ec_839675a54458.slice/crio-dd3ed3ca9cf6770400e6ecba49edf23620e6236fe8a2f9eefbecd03fac955ecc WatchSource:0}: Error finding container dd3ed3ca9cf6770400e6ecba49edf23620e6236fe8a2f9eefbecd03fac955ecc: Status 404 returned error can't find the container with id dd3ed3ca9cf6770400e6ecba49edf23620e6236fe8a2f9eefbecd03fac955ecc
Sep 11 16:15:51.906894 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:51.906836    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:52.217982 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:52.217907    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6" event={"ID":"3aff9c12-7bba-41b8-a7ec-839675a54458","Type":"ContainerStarted","Data":"dd3ed3ca9cf6770400e6ecba49edf23620e6236fe8a2f9eefbecd03fac955ecc"}
Sep 11 16:15:52.238841 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:52.238811    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:52.354376 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:52.354345    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:52.741457 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:52.741425    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:53.224665 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:53.224632    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6" event={"ID":"3aff9c12-7bba-41b8-a7ec-839675a54458","Type":"ContainerStarted","Data":"38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8"}
Sep 11 16:15:53.224834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:53.224732    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:53.302644 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:53.302594    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6" podStartSLOduration=2.8293949290000002 podStartE2EDuration="3.302576892s" totalImagesPullingTime="473.181963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:53.300044674 +0000 UTC m=+1247.018019793" watchObservedRunningTime="2026-09-11 16:15:53.302576892 +0000 UTC m=+1247.020552013"
Sep 11 16:15:53.679150 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:53.679108    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:54.820391 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:54.820357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:55.920066 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:55.920031    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:56.098424 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:56.098387    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:57.555667 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.555628    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"]
Sep 11 16:15:57.559118 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.559095    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.562219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.562066    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-1e0e69-serving-cert\""
Sep 11 16:15:57.574958 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.574934    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"]
Sep 11 16:15:57.622257 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.622214    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:57.721479 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.721422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:57.723614 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.723596    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csvdv\" (UniqueName: \"kubernetes.io/projected/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-kube-api-access-csvdv\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.723677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.723628    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.723677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.723644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.723744 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.723715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.821373 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.821338    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:57.824392 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.824372    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csvdv\" (UniqueName: \"kubernetes.io/projected/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-kube-api-access-csvdv\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.824476 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.824407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.824476 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.824426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.824476 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.824463    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.825029 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.825006    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.826681 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.826656    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.826760 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.826710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.833383 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.833363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csvdv\" (UniqueName: \"kubernetes.io/projected/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-kube-api-access-csvdv\") pod \"maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:57.870174 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:57.870149    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:58.154462 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.154433    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"]
Sep 11 16:15:58.156348 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:15:58.156315    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9f8e2b_6f88_47d1_93cc_89a7a8a97899.slice/crio-bdd73acd6227b9b6cb881e035136af0516e55a465b68f51c151de6f1ffdbbb62 WatchSource:0}: Error finding container bdd73acd6227b9b6cb881e035136af0516e55a465b68f51c151de6f1ffdbbb62: Status 404 returned error can't find the container with id bdd73acd6227b9b6cb881e035136af0516e55a465b68f51c151de6f1ffdbbb62
Sep 11 16:15:58.177641 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.177600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:58.247878 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.247847    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw" event={"ID":"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899","Type":"ContainerStarted","Data":"bdd73acd6227b9b6cb881e035136af0516e55a465b68f51c151de6f1ffdbbb62"}
Sep 11 16:15:58.281096 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.281061    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"]
Sep 11 16:15:58.282176 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.282150    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:58.285475 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.285456    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:15:58.288964 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.288947    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1e0e69-dockercfg-vxbm7\""
Sep 11 16:15:58.289571 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.289556    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1e0e69\""
Sep 11 16:15:58.303255 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.303221    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"]
Sep 11 16:15:58.431017 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.430934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7hd\" (UniqueName: \"kubernetes.io/projected/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-kube-api-access-5p7hd\") pod \"payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t\" (UID: \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:15:58.431017 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.430995    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t\" (UID: \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:15:58.531918 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.531890    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7hd\" (UniqueName: \"kubernetes.io/projected/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-kube-api-access-5p7hd\") pod \"payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t\" (UID: \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:15:58.532072 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.531936    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t\" (UID: \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:15:58.532634 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.532612    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t\" (UID: \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:15:58.542088 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.542066    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7hd\" (UniqueName: \"kubernetes.io/projected/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-kube-api-access-5p7hd\") pod \"payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t\" (UID: \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:15:58.595641 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.595617    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:15:58.661692 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.661634    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:15:58.871366 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:58.871338    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"]
Sep 11 16:15:58.873327 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:15:58.873297    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ad0afd_ab96_4e44_9abf_a2b307dc6657.slice/crio-b6dc325925a1f1c00c8a8b76102433f5995b754358488d4ffe2b324436515a08 WatchSource:0}: Error finding container b6dc325925a1f1c00c8a8b76102433f5995b754358488d4ffe2b324436515a08: Status 404 returned error can't find the container with id b6dc325925a1f1c00c8a8b76102433f5995b754358488d4ffe2b324436515a08
Sep 11 16:15:59.234571 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:59.234461    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:15:59.254683 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:59.254649    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t" event={"ID":"c1ad0afd-ab96-4e44-9abf-a2b307dc6657","Type":"ContainerStarted","Data":"a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36"}
Sep 11 16:15:59.254849 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:59.254714    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:15:59.254849 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:59.254740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t" event={"ID":"c1ad0afd-ab96-4e44-9abf-a2b307dc6657","Type":"ContainerStarted","Data":"b6dc325925a1f1c00c8a8b76102433f5995b754358488d4ffe2b324436515a08"}
Sep 11 16:15:59.256350 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:59.256328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw" event={"ID":"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899","Type":"ContainerStarted","Data":"992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a"}
Sep 11 16:15:59.256475 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:59.256400    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:15:59.283814 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:59.283766    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw" podStartSLOduration=1.901574969 podStartE2EDuration="2.283751317s" totalImagesPullingTime="382.176348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:59.281868987 +0000 UTC m=+1252.999844125" watchObservedRunningTime="2026-09-11 16:15:59.283751317 +0000 UTC m=+1253.001726437"
Sep 11 16:15:59.636806 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:15:59.636769    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:00.444468 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:00.444437    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:01.335999 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:01.335968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:02.431652 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:02.431614    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:05.266398 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:05.266367    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:16:05.287792 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:05.287746    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t" podStartSLOduration=7.287731572 podStartE2EDuration="7.287731572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:15:59.320391219 +0000 UTC m=+1253.038366352" watchObservedRunningTime="2026-09-11 16:16:05.287731572 +0000 UTC m=+1259.005706696"
Sep 11 16:16:10.262986 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:10.262956    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:16:12.286708 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:12.286669    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:14.106897 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:14.106863    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:14.966560 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:14.966528    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:16.069885 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:16.069852    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:17.001751 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:17.001707    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:17.151150 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:17.151107    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:18.057616 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:18.057578    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:18.953252 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:18.953215    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:20.074458 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:20.074425    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:20.867495 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:20.867466    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:20.965874 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:20.965844    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:22.201447 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:22.201408    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:23.086860 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:23.086827    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:26.393049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:26.393007    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:26.800483 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:26.800404    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:27.035646 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:27.035615    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:29.381753 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:29.381716    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:30.062159 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:30.062121    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:30.157435 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:30.157404    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:30.411397 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:30.411325    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:31.544398 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:31.544364    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:32.546259 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:32.546209    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:34.082963 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:34.082929    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:35.099285 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.099234    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"]
Sep 11 16:16:35.104164 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.104142    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.106893 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.106870    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9da1d032-serving-cert\""
Sep 11 16:16:35.118491 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.118465    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"]
Sep 11 16:16:35.140974 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.140951    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:35.198251 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.198221    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-metrics-certs\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.198370 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.198290    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c42db-7724-4d79-afe5-75de8e0fa635-openshift-service-ca\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.198412 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.198372    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.198412 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.198395    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfz66\" (UniqueName: \"kubernetes.io/projected/7d6c42db-7724-4d79-afe5-75de8e0fa635-kube-api-access-rfz66\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.296826 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.296803    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:35.298904 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.298880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.299012 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.298914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfz66\" (UniqueName: \"kubernetes.io/projected/7d6c42db-7724-4d79-afe5-75de8e0fa635-kube-api-access-rfz66\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.299012 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.298966    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-metrics-certs\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.299012 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:16:35.299005    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-serving-cert: secret "maas-api-e2e-mt-9da1d032-serving-cert" not found
Sep 11 16:16:35.299165 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.299031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c42db-7724-4d79-afe5-75de8e0fa635-openshift-service-ca\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.299165 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:16:35.299077    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls podName:7d6c42db-7724-4d79-afe5-75de8e0fa635 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:35.799056674 +0000 UTC m=+1289.517031780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls") pod "maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95" (UID: "7d6c42db-7724-4d79-afe5-75de8e0fa635") : secret "maas-api-e2e-mt-9da1d032-serving-cert" not found
Sep 11 16:16:35.299688 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.299667    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c42db-7724-4d79-afe5-75de8e0fa635-openshift-service-ca\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.301532 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.301512    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-metrics-certs\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.312063 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.312041    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfz66\" (UniqueName: \"kubernetes.io/projected/7d6c42db-7724-4d79-afe5-75de8e0fa635-kube-api-access-rfz66\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.722994 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.722961    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:35.803231 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.803203    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:35.803414 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:16:35.803375    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-serving-cert: secret "maas-api-e2e-mt-9da1d032-serving-cert" not found
Sep 11 16:16:35.803484 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:16:35.803446    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls podName:7d6c42db-7724-4d79-afe5-75de8e0fa635 nodeName:}" failed. No retries permitted until 2026-09-11 16:16:36.803426964 +0000 UTC m=+1290.521402078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls") pod "maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95" (UID: "7d6c42db-7724-4d79-afe5-75de8e0fa635") : secret "maas-api-e2e-mt-9da1d032-serving-cert" not found
Sep 11 16:16:35.832494 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.832462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:35.940424 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:35.940398    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:36.814425 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:36.814394    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:36.816761 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:36.816738    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls\") pod \"maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:36.914207 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:36.914180    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:37.192632 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:37.192602    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"]
Sep 11 16:16:37.193974 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:16:37.193945    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6c42db_7724_4d79_afe5_75de8e0fa635.slice/crio-5b98801c9b94b5fcfcedae46a7fb632742f1c631d726c0ba37d5eaf837edeb01 WatchSource:0}: Error finding container 5b98801c9b94b5fcfcedae46a7fb632742f1c631d726c0ba37d5eaf837edeb01: Status 404 returned error can't find the container with id 5b98801c9b94b5fcfcedae46a7fb632742f1c631d726c0ba37d5eaf837edeb01
Sep 11 16:16:37.441324 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:37.441297    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95" event={"ID":"7d6c42db-7724-4d79-afe5-75de8e0fa635","Type":"ContainerStarted","Data":"5b98801c9b94b5fcfcedae46a7fb632742f1c631d726c0ba37d5eaf837edeb01"}
Sep 11 16:16:38.446940 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:38.446905    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95" event={"ID":"7d6c42db-7724-4d79-afe5-75de8e0fa635","Type":"ContainerStarted","Data":"04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a"}
Sep 11 16:16:38.447321 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:38.446971    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:38.468248 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:38.468194    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95" podStartSLOduration=3.060455552 podStartE2EDuration="3.468166709s" totalImagesPullingTime="407.711157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:16:38.466461004 +0000 UTC m=+1292.184436134" watchObservedRunningTime="2026-09-11 16:16:38.468166709 +0000 UTC m=+1292.186142102"
Sep 11 16:16:44.457703 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:44.457666    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:16:46.734163 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:46.734116    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:16:59.118151 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:16:59.118101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:00.077902 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:00.077868    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:00.178225 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:00.178198    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:01.117547 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:01.117501    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:02.153944 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:02.153908    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:03.359825 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.359790    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:03.768554 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.768478    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"]
Sep 11 16:17:03.772908 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.772889    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:03.780664 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.780647    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9da1d032\"/\"openshift-service-ca.crt\""
Sep 11 16:17:03.781957 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.781940    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9da1d032\"/\"e2e-lifecycle-model-9da1d032-kserve-self-signed-certs\""
Sep 11 16:17:03.782077 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.781980    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-9da1d032\"/\"kube-root-ca.crt\""
Sep 11 16:17:03.782974 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.782954    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-9da1d032\"/\"default-dockercfg-8hqnp\""
Sep 11 16:17:03.796072 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.796050    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"]
Sep 11 16:17:03.803500 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.803477    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:03.930091 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.930064    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-home\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:03.930218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.930107    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tls-certs\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:03.930218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.930138    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdn4\" (UniqueName: \"kubernetes.io/projected/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-kube-api-access-hvdn4\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:03.930218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.930200    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-model-cache\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:03.930362 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.930225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tmp-dir\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:03.930362 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:03.930282    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-dshm\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.031048 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.030973    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-home\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.031048 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.031024    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tls-certs\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.031199 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.031054    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdn4\" (UniqueName: \"kubernetes.io/projected/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-kube-api-access-hvdn4\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.031199 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.031111    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-model-cache\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.031328 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.031230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tmp-dir\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.031328 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.031320    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-dshm\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.031470 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.031450    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-model-cache\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.031537 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.031470    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-home\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.031623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.031603    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tmp-dir\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.033364 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.033345    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-dshm\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.033503 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.033488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tls-certs\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.040665 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.040640    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdn4\" (UniqueName: \"kubernetes.io/projected/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-kube-api-access-hvdn4\") pod \"e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") " pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.082419 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.082397    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:04.354658 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.354634    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"]
Sep 11 16:17:04.356918 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:17:04.356881    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42a3d256_bbfa_4d4d_9f97_05a8b9b0a1fd.slice/crio-4fa092e970cbb23519bcddf991e90f489dee6109bcfc8595052c34758b3c827e WatchSource:0}: Error finding container 4fa092e970cbb23519bcddf991e90f489dee6109bcfc8595052c34758b3c827e: Status 404 returned error can't find the container with id 4fa092e970cbb23519bcddf991e90f489dee6109bcfc8595052c34758b3c827e
Sep 11 16:17:04.370856 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.370829    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:04.568076 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.568040    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px" event={"ID":"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd","Type":"ContainerStarted","Data":"9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476"}
Sep 11 16:17:04.568257 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.568085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px" event={"ID":"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd","Type":"ContainerStarted","Data":"4fa092e970cbb23519bcddf991e90f489dee6109bcfc8595052c34758b3c827e"}
Sep 11 16:17:04.589044 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:04.588995    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px" podStartSLOduration=1.588978367 podStartE2EDuration="1.588978367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:17:04.587176052 +0000 UTC m=+1318.305151172" watchObservedRunningTime="2026-09-11 16:17:04.588978367 +0000 UTC m=+1318.306953487"
Sep 11 16:17:14.082645 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:14.082603    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:14.082645 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:14.082647    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:14.095176 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:14.095151    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:14.624212 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:14.624185    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:17:17.167074 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:17.167044    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:27.151601 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:27.151566    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:45.499780 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:45.499747    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:46.375266 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:46.375212    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:47.285632 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:47.285596    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:48.167722 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:48.167688    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:49.722541 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:49.722509    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:17:50.787957 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:17:50.787923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:00.728298 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:00.728262    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"]
Sep 11 16:18:00.728688 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:00.728493    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95" podUID="7d6c42db-7724-4d79-afe5-75de8e0fa635" containerName="maas-api" containerID="cri-o://04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a" gracePeriod=30
Sep 11 16:18:00.779218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:00.779193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:00.984231 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:00.984174    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:18:01.072144 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.072114    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls\") pod \"7d6c42db-7724-4d79-afe5-75de8e0fa635\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") "
Sep 11 16:18:01.072311 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.072194    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-metrics-certs\") pod \"7d6c42db-7724-4d79-afe5-75de8e0fa635\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") "
Sep 11 16:18:01.072311 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.072283    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c42db-7724-4d79-afe5-75de8e0fa635-openshift-service-ca\") pod \"7d6c42db-7724-4d79-afe5-75de8e0fa635\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") "
Sep 11 16:18:01.072441 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.072331    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfz66\" (UniqueName: \"kubernetes.io/projected/7d6c42db-7724-4d79-afe5-75de8e0fa635-kube-api-access-rfz66\") pod \"7d6c42db-7724-4d79-afe5-75de8e0fa635\" (UID: \"7d6c42db-7724-4d79-afe5-75de8e0fa635\") "
Sep 11 16:18:01.072638 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.072605    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6c42db-7724-4d79-afe5-75de8e0fa635-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d6c42db-7724-4d79-afe5-75de8e0fa635" (UID: "7d6c42db-7724-4d79-afe5-75de8e0fa635"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:18:01.074280 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.074231    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7d6c42db-7724-4d79-afe5-75de8e0fa635" (UID: "7d6c42db-7724-4d79-afe5-75de8e0fa635"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:18:01.074386 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.074295    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d6c42db-7724-4d79-afe5-75de8e0fa635" (UID: "7d6c42db-7724-4d79-afe5-75de8e0fa635"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:18:01.074386 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.074348    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6c42db-7724-4d79-afe5-75de8e0fa635-kube-api-access-rfz66" (OuterVolumeSpecName: "kube-api-access-rfz66") pod "7d6c42db-7724-4d79-afe5-75de8e0fa635" (UID: "7d6c42db-7724-4d79-afe5-75de8e0fa635"). InnerVolumeSpecName "kube-api-access-rfz66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:01.138115 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.138083    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:01.173871 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.173843    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfz66\" (UniqueName: \"kubernetes.io/projected/7d6c42db-7724-4d79-afe5-75de8e0fa635-kube-api-access-rfz66\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:01.173871 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.173868    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:01.174012 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.173878    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d6c42db-7724-4d79-afe5-75de8e0fa635-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:01.174012 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.173886    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d6c42db-7724-4d79-afe5-75de8e0fa635-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:01.402457 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.402407    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:01.523962 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.523931    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:01.842966 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.842935    2564 generic.go:356] "Generic (PLEG): container finished" podID="7d6c42db-7724-4d79-afe5-75de8e0fa635" containerID="04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a" exitCode=0
Sep 11 16:18:01.843383 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.842980    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"
Sep 11 16:18:01.843383 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.843022    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95" event={"ID":"7d6c42db-7724-4d79-afe5-75de8e0fa635","Type":"ContainerDied","Data":"04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a"}
Sep 11 16:18:01.843383 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.843072    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95" event={"ID":"7d6c42db-7724-4d79-afe5-75de8e0fa635","Type":"ContainerDied","Data":"5b98801c9b94b5fcfcedae46a7fb632742f1c631d726c0ba37d5eaf837edeb01"}
Sep 11 16:18:01.843383 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.843093    2564 scope.go:117] "RemoveContainer" containerID="04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a"
Sep 11 16:18:01.852960 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.852926    2564 scope.go:117] "RemoveContainer" containerID="04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a"
Sep 11 16:18:01.853253 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:18:01.853215    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a\": container with ID starting with 04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a not found: ID does not exist" containerID="04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a"
Sep 11 16:18:01.853380 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.853275    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a"} err="failed to get container status \"04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a\": rpc error: code = NotFound desc = could not find container \"04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a\": container with ID starting with 04a78ec70edae6ae2c447e0f6f64345cc508ac487140e1c87d869d2f9252a21a not found: ID does not exist"
Sep 11 16:18:01.872063 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.872039    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"]
Sep 11 16:18:01.878277 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:01.878254    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9da1d032-664bcf78fd-7sg95"]
Sep 11 16:18:02.047601 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:02.047573    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:02.811879 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:02.811845    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d6c42db-7724-4d79-afe5-75de8e0fa635" path="/var/lib/kubelet/pods/7d6c42db-7724-4d79-afe5-75de8e0fa635/volumes"
Sep 11 16:18:03.251218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:03.251123    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:04.474025 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:04.473988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:04.918351 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:04.918306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:05.772182 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:05.772127    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:06.682568 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:06.682520    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:09.924457 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:09.924425    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"]
Sep 11 16:18:10.025868 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:10.025838    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:10.611093 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:10.611054    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:10.712032 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:10.711998    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:24.942648 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:24.942594    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px" podUID="42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" containerName="main" containerID="cri-o://9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476" gracePeriod=45
Sep 11 16:18:25.195826 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.195769    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:18:25.352878 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.352848    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvdn4\" (UniqueName: \"kubernetes.io/projected/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-kube-api-access-hvdn4\") pod \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") "
Sep 11 16:18:25.353053 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.352912    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-dshm\") pod \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") "
Sep 11 16:18:25.353053 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.352953    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-model-cache\") pod \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") "
Sep 11 16:18:25.353053 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.352973    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-home\") pod \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") "
Sep 11 16:18:25.353053 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.353009    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tmp-dir\") pod \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") "
Sep 11 16:18:25.353053 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.353037    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tls-certs\") pod \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\" (UID: \"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd\") "
Sep 11 16:18:25.353360 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.353225    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-home" (OuterVolumeSpecName: "home") pod "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" (UID: "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:18:25.353360 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.353256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" (UID: "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:18:25.353360 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.353339    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-model-cache\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:25.353573 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.353361    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-home\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:25.353573 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.353389    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" (UID: "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:18:25.355096 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.355068    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-dshm" (OuterVolumeSpecName: "dshm") pod "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" (UID: "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:18:25.355208 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.355168    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-kube-api-access-hvdn4" (OuterVolumeSpecName: "kube-api-access-hvdn4") pod "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" (UID: "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd"). InnerVolumeSpecName "kube-api-access-hvdn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:18:25.355463 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.355440    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" (UID: "42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:18:25.454290 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.454207    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tmp-dir\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:25.454290 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.454259    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-tls-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:25.454290 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.454274    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvdn4\" (UniqueName: \"kubernetes.io/projected/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-kube-api-access-hvdn4\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:25.454290 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.454286    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd-dshm\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:18:25.961849 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.961816    2564 generic.go:356] "Generic (PLEG): container finished" podID="42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" containerID="9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476" exitCode=0
Sep 11 16:18:25.962274 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.961869    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"
Sep 11 16:18:25.962274 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.961894    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px" event={"ID":"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd","Type":"ContainerDied","Data":"9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476"}
Sep 11 16:18:25.962274 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.961938    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px" event={"ID":"42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd","Type":"ContainerDied","Data":"4fa092e970cbb23519bcddf991e90f489dee6109bcfc8595052c34758b3c827e"}
Sep 11 16:18:25.962274 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.961959    2564 scope.go:117] "RemoveContainer" containerID="9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476"
Sep 11 16:18:25.971315 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.971293    2564 scope.go:117] "RemoveContainer" containerID="9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476"
Sep 11 16:18:25.971593 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:18:25.971574    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476\": container with ID starting with 9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476 not found: ID does not exist" containerID="9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476"
Sep 11 16:18:25.971658 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.971603    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476"} err="failed to get container status \"9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476\": rpc error: code = NotFound desc = could not find container \"9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476\": container with ID starting with 9211969b8fa1f628fabd6515188e6a2c5a997302b1ed450d783edccccdcd8476 not found: ID does not exist"
Sep 11 16:18:25.995319 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:25.995298    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"]
Sep 11 16:18:26.004041 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:26.004020    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-9da1d032/e2e-lifecycle-model-9da1d032-kserve-c56d6c6d7-f88px"]
Sep 11 16:18:26.812054 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:26.812019    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" path="/var/lib/kubelet/pods/42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd/volumes"
Sep 11 16:18:27.237381 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:27.237306    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:32.095402 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:32.095365    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:32.175321 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:32.175292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:46.280254 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:46.280214    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:48.144824 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:48.144789    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:18:54.705398 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:18:54.705367    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:00.673083 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:00.672996    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:00.768264 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:00.768210    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:00.955158 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:00.955072    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:02.233670 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:02.233632    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:02.342812 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:02.342784    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:02.442640 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:02.442611    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:03.151566 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:03.151533    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:03.359807 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:03.359779    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:04.738888 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:04.738809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:10.927841 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:10.927807    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:11.116279 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:11.116229    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:11.616269 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:11.616221    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:18.848463 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:18.848425    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:18.934302 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:18.934273    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:19.063398 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:19.063364    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:19.222098 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:19.222026    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:29.678409 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:29.678375    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:29.918892 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:29.918858    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:36.478605 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:36.478570    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:37.625184 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:37.625154    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:40.620338 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.620306    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"]
Sep 11 16:19:40.620701 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.620687    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d6c42db-7724-4d79-afe5-75de8e0fa635" containerName="maas-api"
Sep 11 16:19:40.620701 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.620700    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d6c42db-7724-4d79-afe5-75de8e0fa635" containerName="maas-api"
Sep 11 16:19:40.620778 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.620721    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" containerName="main"
Sep 11 16:19:40.620778 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.620727    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" containerName="main"
Sep 11 16:19:40.620778 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.620776    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d6c42db-7724-4d79-afe5-75de8e0fa635" containerName="maas-api"
Sep 11 16:19:40.620871 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.620785    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a3d256-bbfa-4d4d-9f97-05a8b9b0a1fd" containerName="main"
Sep 11 16:19:40.623935 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.623916    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.631851 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.631830    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"sub-test-model-59f108-kserve-self-signed-certs\""
Sep 11 16:19:40.632006 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.631926    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"default-dockercfg-9kwxq\""
Sep 11 16:19:40.632006 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.631939    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"kube-root-ca.crt\""
Sep 11 16:19:40.632146 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.632050    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-59f108\"/\"openshift-service-ca.crt\""
Sep 11 16:19:40.643127 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.643105    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"]
Sep 11 16:19:40.656626 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.656595    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:40.721802 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.721780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tmp-dir\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.721901 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.721808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-home\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.721901 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.721823    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-dshm\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.721901 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.721840    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-model-cache\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.722024 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.721907    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2z5p\" (UniqueName: \"kubernetes.io/projected/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-kube-api-access-d2z5p\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.722024 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.721957    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tls-certs\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.822518 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.822495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tmp-dir\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.822668 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.822525    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-home\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.822668 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.822549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-dshm\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.822668 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.822566    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-model-cache\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.822668 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.822591    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2z5p\" (UniqueName: \"kubernetes.io/projected/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-kube-api-access-d2z5p\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.822668 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.822624    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tls-certs\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.822943 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.822924    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-home\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.823008 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.822990    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-model-cache\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.823055 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.823005    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tmp-dir\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.824663 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.824641    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-dshm\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.824968 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.824949    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tls-certs\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.832159 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.832140    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2z5p\" (UniqueName: \"kubernetes.io/projected/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-kube-api-access-d2z5p\") pod \"sub-test-model-59f108-kserve-59cd7cb97c-2vd5k\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") " pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:40.933268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:40.933188    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:41.224176 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:41.224151    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"]
Sep 11 16:19:41.226392 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:19:41.226361    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3e30f57_0c8b_4918_9d04_1d01db49b2f6.slice/crio-a8989ba669ce99c2dae78359dd0d201b7a8101695951412f44d5c8d50058284f WatchSource:0}: Error finding container a8989ba669ce99c2dae78359dd0d201b7a8101695951412f44d5c8d50058284f: Status 404 returned error can't find the container with id a8989ba669ce99c2dae78359dd0d201b7a8101695951412f44d5c8d50058284f
Sep 11 16:19:41.303118 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:41.303092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k" event={"ID":"e3e30f57-0c8b-4918-9d04-1d01db49b2f6","Type":"ContainerStarted","Data":"9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459"}
Sep 11 16:19:41.303226 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:41.303130    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k" event={"ID":"e3e30f57-0c8b-4918-9d04-1d01db49b2f6","Type":"ContainerStarted","Data":"a8989ba669ce99c2dae78359dd0d201b7a8101695951412f44d5c8d50058284f"}
Sep 11 16:19:41.334926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:41.334848    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k" podStartSLOduration=1.334827122 podStartE2EDuration="1.334827122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:19:41.332685404 +0000 UTC m=+1475.050660545" watchObservedRunningTime="2026-09-11 16:19:41.334827122 +0000 UTC m=+1475.052802246"
Sep 11 16:19:50.934255 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:50.934220    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:50.934648 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:50.934278    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:50.946535 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:50.946510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:51.358873 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:51.358839    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:19:51.806968 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:51.806888    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:58.041131 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:58.041092    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:58.198802 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:58.198772    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:58.403003 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:58.402976    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:58.769166 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:58.769073    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:19:59.039705 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:19:59.039610    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:01.532800 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:01.532767    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:01.658936 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:01.658902    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:01.761797 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:01.761768    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:02.663285 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:02.663247    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:04.143022 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:04.142984    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:05.073550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:05.073516    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:05.216636 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:05.216604    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:06.346680 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.346646    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"]
Sep 11 16:20:06.350142 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.350122    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.353363 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.353339    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c98185-serving-cert\""
Sep 11 16:20:06.418257 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.418221    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:06.454926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.454899    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"]
Sep 11 16:20:06.533704 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.533676    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdxz2\" (UniqueName: \"kubernetes.io/projected/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-kube-api-access-qdxz2\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.533842 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.533721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-metrics-certs\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.533842 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.533764    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-maas-api-tls\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.533842 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.533829    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-openshift-service-ca\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.595729 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.595703    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:06.634989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.634933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-openshift-service-ca\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.634989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.634983    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdxz2\" (UniqueName: \"kubernetes.io/projected/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-kube-api-access-qdxz2\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.635135 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.635012    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-metrics-certs\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.635135 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.635038    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-maas-api-tls\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.635577 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.635554    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-openshift-service-ca\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.637521 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.637502    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-maas-api-tls\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.637598 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.637544    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-metrics-certs\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.649959 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.649931    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdxz2\" (UniqueName: \"kubernetes.io/projected/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-kube-api-access-qdxz2\") pod \"maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.660793 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.660772    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:06.957379 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.957357    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"]
Sep 11 16:20:06.959829 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:20:06.959796    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe69b57_746e_4f4a_b1a0_26f56c1bfe42.slice/crio-cc88d3a67db7ac56132649d4d1f70b037f82d41a4bd34bd3105762c7ed6a17bd WatchSource:0}: Error finding container cc88d3a67db7ac56132649d4d1f70b037f82d41a4bd34bd3105762c7ed6a17bd: Status 404 returned error can't find the container with id cc88d3a67db7ac56132649d4d1f70b037f82d41a4bd34bd3105762c7ed6a17bd
Sep 11 16:20:06.961162 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:06.961141    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:20:07.058739 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:07.058704    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:07.155194 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:07.155161    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:07.424124 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:07.424098    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8" event={"ID":"efe69b57-746e-4f4a-b1a0-26f56c1bfe42","Type":"ContainerStarted","Data":"6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06"}
Sep 11 16:20:07.424524 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:07.424144    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8" event={"ID":"efe69b57-746e-4f4a-b1a0-26f56c1bfe42","Type":"ContainerStarted","Data":"cc88d3a67db7ac56132649d4d1f70b037f82d41a4bd34bd3105762c7ed6a17bd"}
Sep 11 16:20:07.424524 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:07.424225    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:07.446646 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:07.446600    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8" podStartSLOduration=1.0642205869999999 podStartE2EDuration="1.446586204s" totalImagesPullingTime="382.365617ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:07.444540399 +0000 UTC m=+1501.162515554" watchObservedRunningTime="2026-09-11 16:20:07.446586204 +0000 UTC m=+1501.164561323"
Sep 11 16:20:13.434672 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:13.434641    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:20:23.262393 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:23.262352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:24.828007 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:24.827968    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:30.254816 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:30.254739    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:31.701423 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:31.701385    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:31.905218 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:31.905187    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:32.510446 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:32.510413    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:32.876451 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:32.876417    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:33.026036 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:33.026004    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:33.761944 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:33.761897    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:34.395319 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:34.395280    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:35.726666 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:35.726635    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:36.166962 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:36.166930    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:36.356403 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:36.356369    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:37.187434 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:37.187403    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:38.353764 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.353730    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"]
Sep 11 16:20:38.356071 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.356055    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.364060 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.364044    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bf0614-serving-cert\""
Sep 11 16:20:38.385287 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.382727    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"]
Sep 11 16:20:38.423340 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.423317    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:38.469172 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.469140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098a3604-3f81-4ee3-87c3-70f8443fd199-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.469393 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.469261    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-maas-api-tls\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.469393 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.469323    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-metrics-certs\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.469393 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.469365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75q8x\" (UniqueName: \"kubernetes.io/projected/098a3604-3f81-4ee3-87c3-70f8443fd199-kube-api-access-75q8x\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.520116 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.520084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:38.570277 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.570235    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098a3604-3f81-4ee3-87c3-70f8443fd199-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.570408 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.570305    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-maas-api-tls\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.570408 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.570326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-metrics-certs\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.570408 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.570350    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75q8x\" (UniqueName: \"kubernetes.io/projected/098a3604-3f81-4ee3-87c3-70f8443fd199-kube-api-access-75q8x\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.570911 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.570888    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098a3604-3f81-4ee3-87c3-70f8443fd199-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.572851 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.572830    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-maas-api-tls\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.572921 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.572836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-metrics-certs\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.579108 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.579087    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75q8x\" (UniqueName: \"kubernetes.io/projected/098a3604-3f81-4ee3-87c3-70f8443fd199-kube-api-access-75q8x\") pod \"maas-api-e2e-api-b-bf0614-677d6696bc-z46bt\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.619947 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.619887    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:38.666673 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.666651    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:38.991096 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:38.991074    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"]
Sep 11 16:20:38.993250 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:20:38.993201    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098a3604_3f81_4ee3_87c3_70f8443fd199.slice/crio-17830c95e24a1b375ae8f272514502a0f46bb1aac713015fd670ed980fd381ba WatchSource:0}: Error finding container 17830c95e24a1b375ae8f272514502a0f46bb1aac713015fd670ed980fd381ba: Status 404 returned error can't find the container with id 17830c95e24a1b375ae8f272514502a0f46bb1aac713015fd670ed980fd381ba
Sep 11 16:20:39.520925 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:39.520886    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:39.576511 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:39.576484    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt" event={"ID":"098a3604-3f81-4ee3-87c3-70f8443fd199","Type":"ContainerStarted","Data":"304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b"}
Sep 11 16:20:39.576643 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:39.576534    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt" event={"ID":"098a3604-3f81-4ee3-87c3-70f8443fd199","Type":"ContainerStarted","Data":"17830c95e24a1b375ae8f272514502a0f46bb1aac713015fd670ed980fd381ba"}
Sep 11 16:20:39.576643 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:39.576592    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:39.756593 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:39.756485    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt" podStartSLOduration=1.38951451 podStartE2EDuration="1.756470588s" totalImagesPullingTime="366.956078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:39.753344542 +0000 UTC m=+1533.471319689" watchObservedRunningTime="2026-09-11 16:20:39.756470588 +0000 UTC m=+1533.474445708"
Sep 11 16:20:39.943018 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:39.942989    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:40.178028 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.177992    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"]
Sep 11 16:20:40.180737 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.180721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:40.189690 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.189665    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-bf0614\""
Sep 11 16:20:40.193324 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.193302    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-bf0614-dockercfg-jkktk\""
Sep 11 16:20:40.208889 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.208858    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"]
Sep 11 16:20:40.286618 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.286593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd294f58-3809-48e7-9ae1-7d5b79335aab-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h\" (UID: \"bd294f58-3809-48e7-9ae1-7d5b79335aab\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:40.286773 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.286663    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c2x2\" (UniqueName: \"kubernetes.io/projected/bd294f58-3809-48e7-9ae1-7d5b79335aab-kube-api-access-4c2x2\") pod \"payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h\" (UID: \"bd294f58-3809-48e7-9ae1-7d5b79335aab\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:40.387799 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.387766    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd294f58-3809-48e7-9ae1-7d5b79335aab-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h\" (UID: \"bd294f58-3809-48e7-9ae1-7d5b79335aab\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:40.387964 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.387837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c2x2\" (UniqueName: \"kubernetes.io/projected/bd294f58-3809-48e7-9ae1-7d5b79335aab-kube-api-access-4c2x2\") pod \"payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h\" (UID: \"bd294f58-3809-48e7-9ae1-7d5b79335aab\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:40.388559 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.388535    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd294f58-3809-48e7-9ae1-7d5b79335aab-plugins-config-volume\") pod \"payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h\" (UID: \"bd294f58-3809-48e7-9ae1-7d5b79335aab\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:40.406930 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.406905    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c2x2\" (UniqueName: \"kubernetes.io/projected/bd294f58-3809-48e7-9ae1-7d5b79335aab-kube-api-access-4c2x2\") pod \"payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h\" (UID: \"bd294f58-3809-48e7-9ae1-7d5b79335aab\") " pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:40.491873 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.491775    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:40.592466 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.592436    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:40.767550 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:40.767526    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"]
Sep 11 16:20:40.769885 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:20:40.769863    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd294f58_3809_48e7_9ae1_7d5b79335aab.slice/crio-277a5cda822ac5610ddd29f517373865ca765f016d5db058c3fb30a2273f3c0a WatchSource:0}: Error finding container 277a5cda822ac5610ddd29f517373865ca765f016d5db058c3fb30a2273f3c0a: Status 404 returned error can't find the container with id 277a5cda822ac5610ddd29f517373865ca765f016d5db058c3fb30a2273f3c0a
Sep 11 16:20:41.587793 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:41.587749    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h" event={"ID":"bd294f58-3809-48e7-9ae1-7d5b79335aab","Type":"ContainerStarted","Data":"385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5"}
Sep 11 16:20:41.587966 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:41.587815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h" event={"ID":"bd294f58-3809-48e7-9ae1-7d5b79335aab","Type":"ContainerStarted","Data":"277a5cda822ac5610ddd29f517373865ca765f016d5db058c3fb30a2273f3c0a"}
Sep 11 16:20:41.587966 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:41.587837    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:41.617607 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:41.617563    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h" podStartSLOduration=1.617544581 podStartE2EDuration="1.617544581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:20:41.615964741 +0000 UTC m=+1535.333940039" watchObservedRunningTime="2026-09-11 16:20:41.617544581 +0000 UTC m=+1535.335519701"
Sep 11 16:20:41.703777 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:41.703745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:42.696559 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:42.696526    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:43.825493 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:43.825458    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:45.228023 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:45.227985    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:45.586697 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:45.586664    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:20:47.928377 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:47.928335    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:49.568529 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:49.568498    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:50.904561 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:50.904528    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:51.089699 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:51.089655    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:52.473220 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:52.473188    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:52.594316 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:52.594287    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:20:54.014726 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:54.014690    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:55.452818 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:55.452785    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:56.789334 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:56.789302    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:57.986614 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:57.986582    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:20:59.272891 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:20:59.272856    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:00.474486 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:00.474449    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:01.762398 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:01.762366    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:03.004589 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:03.004536    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:03.979211 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:03.979179    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:05.152383 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:05.152341    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:07.418098 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:07.418067    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:10.994192 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:10.994158    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"]
Sep 11 16:21:10.994646 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:10.994385    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8" podUID="efe69b57-746e-4f4a-b1a0-26f56c1bfe42" containerName="maas-api" containerID="cri-o://6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06" gracePeriod=30
Sep 11 16:21:11.043883 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:21:11.043847    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe69b57_746e_4f4a_b1a0_26f56c1bfe42.slice/crio-conmon-6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 16:21:11.072548 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.072522    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:11.218977 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.218949    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:11.244926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.244876    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:21:11.312382 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.312347    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:11.346893 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.346870    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-maas-api-tls\") pod \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") "
Sep 11 16:21:11.347016 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.346902    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdxz2\" (UniqueName: \"kubernetes.io/projected/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-kube-api-access-qdxz2\") pod \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") "
Sep 11 16:21:11.347016 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.346938    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-metrics-certs\") pod \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") "
Sep 11 16:21:11.347016 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.347002    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-openshift-service-ca\") pod \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\" (UID: \"efe69b57-746e-4f4a-b1a0-26f56c1bfe42\") "
Sep 11 16:21:11.347415 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.347389    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "efe69b57-746e-4f4a-b1a0-26f56c1bfe42" (UID: "efe69b57-746e-4f4a-b1a0-26f56c1bfe42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:11.349039 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.349012    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "efe69b57-746e-4f4a-b1a0-26f56c1bfe42" (UID: "efe69b57-746e-4f4a-b1a0-26f56c1bfe42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:11.349224 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.349201    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "efe69b57-746e-4f4a-b1a0-26f56c1bfe42" (UID: "efe69b57-746e-4f4a-b1a0-26f56c1bfe42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:11.349297 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.349220    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-kube-api-access-qdxz2" (OuterVolumeSpecName: "kube-api-access-qdxz2") pod "efe69b57-746e-4f4a-b1a0-26f56c1bfe42" (UID: "efe69b57-746e-4f4a-b1a0-26f56c1bfe42"). InnerVolumeSpecName "kube-api-access-qdxz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:11.421534 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.421495    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:11.448266 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.448222    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:11.448377 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.448268    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdxz2\" (UniqueName: \"kubernetes.io/projected/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-kube-api-access-qdxz2\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:11.448377 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.448282    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:11.448377 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.448292    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efe69b57-746e-4f4a-b1a0-26f56c1bfe42-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:11.528295 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.528197    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:11.732019 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.731975    2564 generic.go:356] "Generic (PLEG): container finished" podID="efe69b57-746e-4f4a-b1a0-26f56c1bfe42" containerID="6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06" exitCode=0
Sep 11 16:21:11.732194 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.732021    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"
Sep 11 16:21:11.732194 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.732033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8" event={"ID":"efe69b57-746e-4f4a-b1a0-26f56c1bfe42","Type":"ContainerDied","Data":"6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06"}
Sep 11 16:21:11.732194 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.732085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8" event={"ID":"efe69b57-746e-4f4a-b1a0-26f56c1bfe42","Type":"ContainerDied","Data":"cc88d3a67db7ac56132649d4d1f70b037f82d41a4bd34bd3105762c7ed6a17bd"}
Sep 11 16:21:11.732194 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.732112    2564 scope.go:117] "RemoveContainer" containerID="6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06"
Sep 11 16:21:11.741380 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.741361    2564 scope.go:117] "RemoveContainer" containerID="6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06"
Sep 11 16:21:11.741679 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:21:11.741647    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06\": container with ID starting with 6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06 not found: ID does not exist" containerID="6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06"
Sep 11 16:21:11.741745 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.741689    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06"} err="failed to get container status \"6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06\": rpc error: code = NotFound desc = could not find container \"6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06\": container with ID starting with 6c927e32eb8e6514e442e5d2c4761a243fe8d031f9c85cc5843449fd1677ae06 not found: ID does not exist"
Sep 11 16:21:11.760955 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.760934    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"]
Sep 11 16:21:11.766920 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.766901    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c98185-54d459dcc5-7d7m8"]
Sep 11 16:21:11.810144 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.810094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:11.956977 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:11.956952    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:12.813151 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:12.813106    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efe69b57-746e-4f4a-b1a0-26f56c1bfe42" path="/var/lib/kubelet/pods/efe69b57-746e-4f4a-b1a0-26f56c1bfe42/volumes"
Sep 11 16:21:12.880208 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:12.880176    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:13.936862 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:13.936830    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:14.857524 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:14.857488    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"]
Sep 11 16:21:14.883571 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:14.883541    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:15.154119 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:15.154035    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:15.747555 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:15.747519    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:15.955255 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:15.955218    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:16.756417 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:16.756381    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:17.759119 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:17.759075    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:18.755600 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:18.755566    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:18.900823 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:18.900791    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:19.233551 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:19.233523    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:19.918458 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:19.918425    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:20.888073 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:20.888029    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:21.621437 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:21.621399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:22.552740 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:22.552709    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:23.478105 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:23.478065    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:24.646628 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:24.646600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:25.549422 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:25.549390    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:26.048300 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:26.048271    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:29.875876 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:29.875840    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k" podUID="e3e30f57-0c8b-4918-9d04-1d01db49b2f6" containerName="main" containerID="cri-o://9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459" gracePeriod=45
Sep 11 16:21:30.131073 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.131016    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:21:30.197488 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197465    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-dshm\") pod \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") "
Sep 11 16:21:30.197620 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197499    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2z5p\" (UniqueName: \"kubernetes.io/projected/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-kube-api-access-d2z5p\") pod \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") "
Sep 11 16:21:30.197620 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197551    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tmp-dir\") pod \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") "
Sep 11 16:21:30.197620 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197572    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tls-certs\") pod \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") "
Sep 11 16:21:30.197620 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197590    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-model-cache\") pod \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") "
Sep 11 16:21:30.197826 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197641    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-home\") pod \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\" (UID: \"e3e30f57-0c8b-4918-9d04-1d01db49b2f6\") "
Sep 11 16:21:30.197887 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197840    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3e30f57-0c8b-4918-9d04-1d01db49b2f6" (UID: "e3e30f57-0c8b-4918-9d04-1d01db49b2f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:30.197887 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197854    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3e30f57-0c8b-4918-9d04-1d01db49b2f6" (UID: "e3e30f57-0c8b-4918-9d04-1d01db49b2f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:30.197988 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197917    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-home" (OuterVolumeSpecName: "home") pod "e3e30f57-0c8b-4918-9d04-1d01db49b2f6" (UID: "e3e30f57-0c8b-4918-9d04-1d01db49b2f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:30.197988 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.197950    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tmp-dir\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:30.199597 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.199574    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-dshm" (OuterVolumeSpecName: "dshm") pod "e3e30f57-0c8b-4918-9d04-1d01db49b2f6" (UID: "e3e30f57-0c8b-4918-9d04-1d01db49b2f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:21:30.199702 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.199650    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3e30f57-0c8b-4918-9d04-1d01db49b2f6" (UID: "e3e30f57-0c8b-4918-9d04-1d01db49b2f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:30.199702 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.199667    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-kube-api-access-d2z5p" (OuterVolumeSpecName: "kube-api-access-d2z5p") pod "e3e30f57-0c8b-4918-9d04-1d01db49b2f6" (UID: "e3e30f57-0c8b-4918-9d04-1d01db49b2f6"). InnerVolumeSpecName "kube-api-access-d2z5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:30.299165 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.299138    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-tls-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:30.299165 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.299163    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-model-cache\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:30.299327 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.299178    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-home\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:30.299327 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.299188    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-dshm\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:30.299327 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.299200    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2z5p\" (UniqueName: \"kubernetes.io/projected/e3e30f57-0c8b-4918-9d04-1d01db49b2f6-kube-api-access-d2z5p\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:30.830548 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.830516    2564 generic.go:356] "Generic (PLEG): container finished" podID="e3e30f57-0c8b-4918-9d04-1d01db49b2f6" containerID="9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459" exitCode=0
Sep 11 16:21:30.830677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.830563    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"
Sep 11 16:21:30.830677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.830599    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k" event={"ID":"e3e30f57-0c8b-4918-9d04-1d01db49b2f6","Type":"ContainerDied","Data":"9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459"}
Sep 11 16:21:30.830677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.830649    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k" event={"ID":"e3e30f57-0c8b-4918-9d04-1d01db49b2f6","Type":"ContainerDied","Data":"a8989ba669ce99c2dae78359dd0d201b7a8101695951412f44d5c8d50058284f"}
Sep 11 16:21:30.830677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.830669    2564 scope.go:117] "RemoveContainer" containerID="9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459"
Sep 11 16:21:30.839005 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.838988    2564 scope.go:117] "RemoveContainer" containerID="9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459"
Sep 11 16:21:30.839314 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:21:30.839294    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459\": container with ID starting with 9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459 not found: ID does not exist" containerID="9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459"
Sep 11 16:21:30.839382 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.839323    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459"} err="failed to get container status \"9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459\": rpc error: code = NotFound desc = could not find container \"9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459\": container with ID starting with 9e3d97b1901b8305c1ec6ed9da9259c93b24164b6dba266b4840ee6f4e2d8459 not found: ID does not exist"
Sep 11 16:21:30.848416 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.848394    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"]
Sep 11 16:21:30.852747 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.852725    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-59f108/sub-test-model-59f108-kserve-59cd7cb97c-2vd5k"]
Sep 11 16:21:30.883746 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:30.883727    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:32.810937 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:32.810906    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3e30f57-0c8b-4918-9d04-1d01db49b2f6" path="/var/lib/kubelet/pods/e3e30f57-0c8b-4918-9d04-1d01db49b2f6/volumes"
Sep 11 16:21:35.092874 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.092838    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"]
Sep 11 16:21:35.093374 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.093119    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt" podUID="098a3604-3f81-4ee3-87c3-70f8443fd199" containerName="maas-api" containerID="cri-o://304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b" gracePeriod=30
Sep 11 16:21:35.130047 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.130022    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:35.177346 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.177322    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"]
Sep 11 16:21:35.177581 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.177555    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h" podUID="bd294f58-3809-48e7-9ae1-7d5b79335aab" containerName="payload-processing" containerID="cri-o://385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5" gracePeriod=30
Sep 11 16:21:35.228865 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.228837    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:35.337286 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.337192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:35.426867 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.426843    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:21:35.429455 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.429433    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:35.451451 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.451427    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:21:35.533347 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.533323    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:35.537351 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537328    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c2x2\" (UniqueName: \"kubernetes.io/projected/bd294f58-3809-48e7-9ae1-7d5b79335aab-kube-api-access-4c2x2\") pod \"bd294f58-3809-48e7-9ae1-7d5b79335aab\" (UID: \"bd294f58-3809-48e7-9ae1-7d5b79335aab\") "
Sep 11 16:21:35.537473 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537377    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098a3604-3f81-4ee3-87c3-70f8443fd199-openshift-service-ca\") pod \"098a3604-3f81-4ee3-87c3-70f8443fd199\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") "
Sep 11 16:21:35.537473 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537407    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75q8x\" (UniqueName: \"kubernetes.io/projected/098a3604-3f81-4ee3-87c3-70f8443fd199-kube-api-access-75q8x\") pod \"098a3604-3f81-4ee3-87c3-70f8443fd199\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") "
Sep 11 16:21:35.537473 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537423    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd294f58-3809-48e7-9ae1-7d5b79335aab-plugins-config-volume\") pod \"bd294f58-3809-48e7-9ae1-7d5b79335aab\" (UID: \"bd294f58-3809-48e7-9ae1-7d5b79335aab\") "
Sep 11 16:21:35.537473 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537462    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-maas-api-tls\") pod \"098a3604-3f81-4ee3-87c3-70f8443fd199\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") "
Sep 11 16:21:35.537659 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537539    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-metrics-certs\") pod \"098a3604-3f81-4ee3-87c3-70f8443fd199\" (UID: \"098a3604-3f81-4ee3-87c3-70f8443fd199\") "
Sep 11 16:21:35.537788 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537763    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098a3604-3f81-4ee3-87c3-70f8443fd199-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "098a3604-3f81-4ee3-87c3-70f8443fd199" (UID: "098a3604-3f81-4ee3-87c3-70f8443fd199"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:35.537899 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537822    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd294f58-3809-48e7-9ae1-7d5b79335aab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd294f58-3809-48e7-9ae1-7d5b79335aab" (UID: "bd294f58-3809-48e7-9ae1-7d5b79335aab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:35.537962 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537917    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/098a3604-3f81-4ee3-87c3-70f8443fd199-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:35.537962 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.537937    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd294f58-3809-48e7-9ae1-7d5b79335aab-plugins-config-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:35.539632 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.539602    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "098a3604-3f81-4ee3-87c3-70f8443fd199" (UID: "098a3604-3f81-4ee3-87c3-70f8443fd199"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:35.539632 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.539620    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd294f58-3809-48e7-9ae1-7d5b79335aab-kube-api-access-4c2x2" (OuterVolumeSpecName: "kube-api-access-4c2x2") pod "bd294f58-3809-48e7-9ae1-7d5b79335aab" (UID: "bd294f58-3809-48e7-9ae1-7d5b79335aab"). InnerVolumeSpecName "kube-api-access-4c2x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:35.539912 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.539892    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098a3604-3f81-4ee3-87c3-70f8443fd199-kube-api-access-75q8x" (OuterVolumeSpecName: "kube-api-access-75q8x") pod "098a3604-3f81-4ee3-87c3-70f8443fd199" (UID: "098a3604-3f81-4ee3-87c3-70f8443fd199"). InnerVolumeSpecName "kube-api-access-75q8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:35.539912 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.539898    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "098a3604-3f81-4ee3-87c3-70f8443fd199" (UID: "098a3604-3f81-4ee3-87c3-70f8443fd199"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:35.639124 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.639058    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c2x2\" (UniqueName: \"kubernetes.io/projected/bd294f58-3809-48e7-9ae1-7d5b79335aab-kube-api-access-4c2x2\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:35.639124 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.639081    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75q8x\" (UniqueName: \"kubernetes.io/projected/098a3604-3f81-4ee3-87c3-70f8443fd199-kube-api-access-75q8x\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:35.639124 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.639094    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:35.639124 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.639103    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/098a3604-3f81-4ee3-87c3-70f8443fd199-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:35.854703 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.854667    2564 generic.go:356] "Generic (PLEG): container finished" podID="bd294f58-3809-48e7-9ae1-7d5b79335aab" containerID="385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5" exitCode=0
Sep 11 16:21:35.854859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.854720    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"
Sep 11 16:21:35.854859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.854757    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h" event={"ID":"bd294f58-3809-48e7-9ae1-7d5b79335aab","Type":"ContainerDied","Data":"385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5"}
Sep 11 16:21:35.854859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.854819    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h" event={"ID":"bd294f58-3809-48e7-9ae1-7d5b79335aab","Type":"ContainerDied","Data":"277a5cda822ac5610ddd29f517373865ca765f016d5db058c3fb30a2273f3c0a"}
Sep 11 16:21:35.854859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.854851    2564 scope.go:117] "RemoveContainer" containerID="385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5"
Sep 11 16:21:35.856284 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.856259    2564 generic.go:356] "Generic (PLEG): container finished" podID="098a3604-3f81-4ee3-87c3-70f8443fd199" containerID="304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b" exitCode=0
Sep 11 16:21:35.856407 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.856294    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"
Sep 11 16:21:35.856407 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.856307    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt" event={"ID":"098a3604-3f81-4ee3-87c3-70f8443fd199","Type":"ContainerDied","Data":"304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b"}
Sep 11 16:21:35.856407 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.856350    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt" event={"ID":"098a3604-3f81-4ee3-87c3-70f8443fd199","Type":"ContainerDied","Data":"17830c95e24a1b375ae8f272514502a0f46bb1aac713015fd670ed980fd381ba"}
Sep 11 16:21:35.866277 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.866228    2564 scope.go:117] "RemoveContainer" containerID="385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5"
Sep 11 16:21:35.866712 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:21:35.866683    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5\": container with ID starting with 385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5 not found: ID does not exist" containerID="385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5"
Sep 11 16:21:35.866822 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.866720    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5"} err="failed to get container status \"385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5\": rpc error: code = NotFound desc = could not find container \"385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5\": container with ID starting with 385f68c858964d00df427e8b3acb3998b4fb2c1c3275c94944662774b8859ac5 not found: ID does not exist"
Sep 11 16:21:35.866822 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.866742    2564 scope.go:117] "RemoveContainer" containerID="304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b"
Sep 11 16:21:35.875061 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.875042    2564 scope.go:117] "RemoveContainer" containerID="304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b"
Sep 11 16:21:35.875342 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:21:35.875319    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b\": container with ID starting with 304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b not found: ID does not exist" containerID="304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b"
Sep 11 16:21:35.875422 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.875352    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b"} err="failed to get container status \"304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b\": rpc error: code = NotFound desc = could not find container \"304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b\": container with ID starting with 304c7aad55f026a7e390827fb15a38a04507f3a476a31fd81a4ce2061227eb6b not found: ID does not exist"
Sep 11 16:21:35.878761 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.878741    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:35.888932 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.888908    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"]
Sep 11 16:21:35.893456 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.893410    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf0614-677d6696bc-z46bt"]
Sep 11 16:21:35.913381 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.913339    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"]
Sep 11 16:21:35.918349 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.918331    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-bf0614-7f559684c5-qd56h"]
Sep 11 16:21:35.961232 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:35.961205    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:36.812971 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:36.812932    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="098a3604-3f81-4ee3-87c3-70f8443fd199" path="/var/lib/kubelet/pods/098a3604-3f81-4ee3-87c3-70f8443fd199/volumes"
Sep 11 16:21:36.813477 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:36.813441    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd294f58-3809-48e7-9ae1-7d5b79335aab" path="/var/lib/kubelet/pods/bd294f58-3809-48e7-9ae1-7d5b79335aab/volumes"
Sep 11 16:21:37.050751 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:37.050706    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:38.008959 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:38.008926    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:38.955444 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:38.955401    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:39.855009 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:39.854963    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:42.263306 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:42.263270    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:42.369960 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:42.369922    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:42.572459 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:42.572429    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:43.309480 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:43.309445    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:44.255121 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:44.255088    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:45.124005 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:45.123964    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:45.996259 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:45.996205    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:46.888783 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:46.888751    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:48.213741 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:48.213709    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:53.709028 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.708997    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"]
Sep 11 16:21:53.709455 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.709195    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd" podUID="4affd20a-c6ef-4ffe-825a-c414e9426c45" containerName="maas-api" containerID="cri-o://b7eb958772954c35d92d1b0d54e799d497665243881f32e495cc7be721d94b81" gracePeriod=30
Sep 11 16:21:53.770315 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.770281    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:53.874415 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.874381    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:53.943857 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.943833    2564 generic.go:356] "Generic (PLEG): container finished" podID="4affd20a-c6ef-4ffe-825a-c414e9426c45" containerID="b7eb958772954c35d92d1b0d54e799d497665243881f32e495cc7be721d94b81" exitCode=0
Sep 11 16:21:53.943993 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.943908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd" event={"ID":"4affd20a-c6ef-4ffe-825a-c414e9426c45","Type":"ContainerDied","Data":"b7eb958772954c35d92d1b0d54e799d497665243881f32e495cc7be721d94b81"}
Sep 11 16:21:53.943993 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.943970    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd" event={"ID":"4affd20a-c6ef-4ffe-825a-c414e9426c45","Type":"ContainerDied","Data":"df366535bdfe2d4e3b7ad23e693ec0634851a8cd2c4ea336d52af0d90d4f1ebf"}
Sep 11 16:21:53.943993 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.943990    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df366535bdfe2d4e3b7ad23e693ec0634851a8cd2c4ea336d52af0d90d4f1ebf"
Sep 11 16:21:53.946210 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.946187    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:21:53.980476 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:53.980400    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:54.074182 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.074160    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:54.080314 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.080291    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-maas-api-tls\") pod \"4affd20a-c6ef-4ffe-825a-c414e9426c45\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") "
Sep 11 16:21:54.080415 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.080331    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-metrics-certs\") pod \"4affd20a-c6ef-4ffe-825a-c414e9426c45\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") "
Sep 11 16:21:54.080415 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.080368    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4affd20a-c6ef-4ffe-825a-c414e9426c45-openshift-service-ca\") pod \"4affd20a-c6ef-4ffe-825a-c414e9426c45\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") "
Sep 11 16:21:54.080527 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.080418    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lwx2\" (UniqueName: \"kubernetes.io/projected/4affd20a-c6ef-4ffe-825a-c414e9426c45-kube-api-access-7lwx2\") pod \"4affd20a-c6ef-4ffe-825a-c414e9426c45\" (UID: \"4affd20a-c6ef-4ffe-825a-c414e9426c45\") "
Sep 11 16:21:54.080793 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.080766    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4affd20a-c6ef-4ffe-825a-c414e9426c45-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4affd20a-c6ef-4ffe-825a-c414e9426c45" (UID: "4affd20a-c6ef-4ffe-825a-c414e9426c45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:21:54.082490 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.082461    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4affd20a-c6ef-4ffe-825a-c414e9426c45" (UID: "4affd20a-c6ef-4ffe-825a-c414e9426c45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:54.082571 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.082483    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4affd20a-c6ef-4ffe-825a-c414e9426c45" (UID: "4affd20a-c6ef-4ffe-825a-c414e9426c45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:21:54.082571 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.082530    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4affd20a-c6ef-4ffe-825a-c414e9426c45-kube-api-access-7lwx2" (OuterVolumeSpecName: "kube-api-access-7lwx2") pod "4affd20a-c6ef-4ffe-825a-c414e9426c45" (UID: "4affd20a-c6ef-4ffe-825a-c414e9426c45"). InnerVolumeSpecName "kube-api-access-7lwx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:21:54.181215 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.181183    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:54.181215 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.181209    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4affd20a-c6ef-4ffe-825a-c414e9426c45-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:54.181215 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.181221    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4affd20a-c6ef-4ffe-825a-c414e9426c45-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:54.181514 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.181230    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lwx2\" (UniqueName: \"kubernetes.io/projected/4affd20a-c6ef-4ffe-825a-c414e9426c45-kube-api-access-7lwx2\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:21:54.184767 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.184699    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:54.487036 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.487001    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:54.589394 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.589362    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:54.947993 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.947960    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"
Sep 11 16:21:54.967834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.967805    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"]
Sep 11 16:21:54.971712 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:54.971690    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f7c21d-fbc5c8979-sb5cd"]
Sep 11 16:21:55.451169 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:55.451130    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:56.344022 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:56.343979    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:56.811106 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:56.811076    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4affd20a-c6ef-4ffe-825a-c414e9426c45" path="/var/lib/kubelet/pods/4affd20a-c6ef-4ffe-825a-c414e9426c45/volumes"
Sep 11 16:21:57.286632 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:57.286522    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:58.906444 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906407    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"]
Sep 11 16:21:58.906798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906736    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="098a3604-3f81-4ee3-87c3-70f8443fd199" containerName="maas-api"
Sep 11 16:21:58.906798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906747    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="098a3604-3f81-4ee3-87c3-70f8443fd199" containerName="maas-api"
Sep 11 16:21:58.906798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906761    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4affd20a-c6ef-4ffe-825a-c414e9426c45" containerName="maas-api"
Sep 11 16:21:58.906798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906766    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4affd20a-c6ef-4ffe-825a-c414e9426c45" containerName="maas-api"
Sep 11 16:21:58.906798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906774    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3e30f57-0c8b-4918-9d04-1d01db49b2f6" containerName="main"
Sep 11 16:21:58.906798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906780    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3e30f57-0c8b-4918-9d04-1d01db49b2f6" containerName="main"
Sep 11 16:21:58.906798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906787    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd294f58-3809-48e7-9ae1-7d5b79335aab" containerName="payload-processing"
Sep 11 16:21:58.906798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906793    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd294f58-3809-48e7-9ae1-7d5b79335aab" containerName="payload-processing"
Sep 11 16:21:58.907049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906806    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efe69b57-746e-4f4a-b1a0-26f56c1bfe42" containerName="maas-api"
Sep 11 16:21:58.907049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906811    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efe69b57-746e-4f4a-b1a0-26f56c1bfe42" containerName="maas-api"
Sep 11 16:21:58.907049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906862    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efe69b57-746e-4f4a-b1a0-26f56c1bfe42" containerName="maas-api"
Sep 11 16:21:58.907049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906871    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3e30f57-0c8b-4918-9d04-1d01db49b2f6" containerName="main"
Sep 11 16:21:58.907049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906878    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd294f58-3809-48e7-9ae1-7d5b79335aab" containerName="payload-processing"
Sep 11 16:21:58.907049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906884    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="098a3604-3f81-4ee3-87c3-70f8443fd199" containerName="maas-api"
Sep 11 16:21:58.907049 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.906889    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4affd20a-c6ef-4ffe-825a-c414e9426c45" containerName="maas-api"
Sep 11 16:21:58.910103 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.910081    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:58.915043 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.915025    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"default-dockercfg-ss8lx\""
Sep 11 16:21:58.915139 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.915083    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"openshift-service-ca.crt\""
Sep 11 16:21:58.915217 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.915199    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"rate-test-model-1e0e69-kserve-self-signed-certs\""
Sep 11 16:21:58.915837 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.915818    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-1e0e69\"/\"kube-root-ca.crt\""
Sep 11 16:21:58.928142 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.928119    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"]
Sep 11 16:21:58.956066 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:58.956033    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:21:59.017323 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.017299    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tmp-dir\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.017438 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.017340    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tls-certs\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.017438 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.017408    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-model-cache\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.017555 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.017479    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8frx\" (UniqueName: \"kubernetes.io/projected/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-kube-api-access-t8frx\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.017555 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.017511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-dshm\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.017555 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.017541    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-home\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.118609 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.118550    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8frx\" (UniqueName: \"kubernetes.io/projected/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-kube-api-access-t8frx\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.118609 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.118585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-dshm\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.118609 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.118606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-home\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.118837 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.118659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tmp-dir\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.118837 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.118693    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tls-certs\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.118837 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.118731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-model-cache\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.119074 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.119051    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-home\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.119131 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.119109    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tmp-dir\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.119131 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.119120    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-model-cache\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.120922 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.120899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-dshm\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.121065 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.121049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tls-certs\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.127670 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.127644    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8frx\" (UniqueName: \"kubernetes.io/projected/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-kube-api-access-t8frx\") pod \"rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") " pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.219178 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.219132    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:21:59.496336 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.496306    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"]
Sep 11 16:21:59.497360 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:21:59.497336    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod909c63b7_dcdd_4db2_b9d0_c2a851e561c4.slice/crio-ea77776889f2117cd0ea891fe70f6b68a4d33ff94d519ea40c25daaa2aa2a2c9 WatchSource:0}: Error finding container ea77776889f2117cd0ea891fe70f6b68a4d33ff94d519ea40c25daaa2aa2a2c9: Status 404 returned error can't find the container with id ea77776889f2117cd0ea891fe70f6b68a4d33ff94d519ea40c25daaa2aa2a2c9
Sep 11 16:21:59.971968 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.971937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd" event={"ID":"909c63b7-dcdd-4db2-b9d0-c2a851e561c4","Type":"ContainerStarted","Data":"eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a"}
Sep 11 16:21:59.971968 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.971974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd" event={"ID":"909c63b7-dcdd-4db2-b9d0-c2a851e561c4","Type":"ContainerStarted","Data":"ea77776889f2117cd0ea891fe70f6b68a4d33ff94d519ea40c25daaa2aa2a2c9"}
Sep 11 16:21:59.993570 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:21:59.993528    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd" podStartSLOduration=1.993514088 podStartE2EDuration="1.993514088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:21:59.992151298 +0000 UTC m=+1613.710126428" watchObservedRunningTime="2026-09-11 16:21:59.993514088 +0000 UTC m=+1613.711489208"
Sep 11 16:22:00.775168 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:00.775134    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:00.847774 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:00.847744    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:01.064923 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:01.064892    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:01.845168 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:01.845133    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:02.971866 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:02.971821    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:03.863572 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:03.863528    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:04.920000 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:04.919969    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:05.806436 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:05.806403    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:06.916723 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:06.916695    2564 scope.go:117] "RemoveContainer" containerID="b7eb958772954c35d92d1b0d54e799d497665243881f32e495cc7be721d94b81"
Sep 11 16:22:09.219757 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:09.219721    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:22:09.220122 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:09.219775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:22:09.232151 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:09.232119    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:22:10.034023 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:10.033991    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:22:10.210477 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:10.210445    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:12.115537 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.115504    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"]
Sep 11 16:22:12.115982 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.115732    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6" podUID="3aff9c12-7bba-41b8-a7ec-839675a54458" containerName="maas-api" containerID="cri-o://38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8" gracePeriod=30
Sep 11 16:22:12.147446 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.147410    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:12.224413 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.224345    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:12.302899 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.302874    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:12.362080 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.362048    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:22:12.375657 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.375589    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:12.449421 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.449391    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:12.527123 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.527097    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm5tq\" (UniqueName: \"kubernetes.io/projected/3aff9c12-7bba-41b8-a7ec-839675a54458-kube-api-access-nm5tq\") pod \"3aff9c12-7bba-41b8-a7ec-839675a54458\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") "
Sep 11 16:22:12.527272 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.527161    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-maas-api-tls\") pod \"3aff9c12-7bba-41b8-a7ec-839675a54458\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") "
Sep 11 16:22:12.527329 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.527282    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-metrics-certs\") pod \"3aff9c12-7bba-41b8-a7ec-839675a54458\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") "
Sep 11 16:22:12.527386 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.527367    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff9c12-7bba-41b8-a7ec-839675a54458-openshift-service-ca\") pod \"3aff9c12-7bba-41b8-a7ec-839675a54458\" (UID: \"3aff9c12-7bba-41b8-a7ec-839675a54458\") "
Sep 11 16:22:12.527726 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.527701    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aff9c12-7bba-41b8-a7ec-839675a54458-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3aff9c12-7bba-41b8-a7ec-839675a54458" (UID: "3aff9c12-7bba-41b8-a7ec-839675a54458"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:22:12.529355 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.529322    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3aff9c12-7bba-41b8-a7ec-839675a54458" (UID: "3aff9c12-7bba-41b8-a7ec-839675a54458"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:22:12.529720 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.529692    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3aff9c12-7bba-41b8-a7ec-839675a54458" (UID: "3aff9c12-7bba-41b8-a7ec-839675a54458"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:22:12.529820 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.529755    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aff9c12-7bba-41b8-a7ec-839675a54458-kube-api-access-nm5tq" (OuterVolumeSpecName: "kube-api-access-nm5tq") pod "3aff9c12-7bba-41b8-a7ec-839675a54458" (UID: "3aff9c12-7bba-41b8-a7ec-839675a54458"). InnerVolumeSpecName "kube-api-access-nm5tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:22:12.628627 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.628563    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm5tq\" (UniqueName: \"kubernetes.io/projected/3aff9c12-7bba-41b8-a7ec-839675a54458-kube-api-access-nm5tq\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:12.628627 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.628591    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:12.628627 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.628601    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3aff9c12-7bba-41b8-a7ec-839675a54458-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:12.628627 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.628611    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aff9c12-7bba-41b8-a7ec-839675a54458-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:12.799518 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:12.799481    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:13.037709 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.037636    2564 generic.go:356] "Generic (PLEG): container finished" podID="3aff9c12-7bba-41b8-a7ec-839675a54458" containerID="38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8" exitCode=0
Sep 11 16:22:13.037709 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.037682    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"
Sep 11 16:22:13.037870 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.037722    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6" event={"ID":"3aff9c12-7bba-41b8-a7ec-839675a54458","Type":"ContainerDied","Data":"38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8"}
Sep 11 16:22:13.037870 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.037778    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6" event={"ID":"3aff9c12-7bba-41b8-a7ec-839675a54458","Type":"ContainerDied","Data":"dd3ed3ca9cf6770400e6ecba49edf23620e6236fe8a2f9eefbecd03fac955ecc"}
Sep 11 16:22:13.037870 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.037807    2564 scope.go:117] "RemoveContainer" containerID="38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8"
Sep 11 16:22:13.045494 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.045477    2564 scope.go:117] "RemoveContainer" containerID="38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8"
Sep 11 16:22:13.045767 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:22:13.045750    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8\": container with ID starting with 38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8 not found: ID does not exist" containerID="38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8"
Sep 11 16:22:13.045814 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.045773    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8"} err="failed to get container status \"38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8\": rpc error: code = NotFound desc = could not find container \"38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8\": container with ID starting with 38d766ba403102ec5be7842e7fa1b3590579f2e25c4a88af419d73a97da578b8 not found: ID does not exist"
Sep 11 16:22:13.052179 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.052158    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:13.083628 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.083604    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"]
Sep 11 16:22:13.099222 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.099197    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-320b99-86467fd954-z4qm6"]
Sep 11 16:22:13.889068 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:13.889026    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:14.702031 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:14.701995    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:14.811199 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:14.811164    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3aff9c12-7bba-41b8-a7ec-839675a54458" path="/var/lib/kubelet/pods/3aff9c12-7bba-41b8-a7ec-839675a54458/volumes"
Sep 11 16:22:19.710973 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:19.710945    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:19.814441 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:19.814410    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:20.037177 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:20.037097    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:41.704811 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:41.704774    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:42.635564 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:42.635528    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"]
Sep 11 16:22:42.681375 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:42.681344    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:43.116789 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:43.116763    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:43.493533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:43.493457    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:43.592064 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:43.592025    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:45.953021 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:45.952988    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:46.761478 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:46.761439    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:46.909131 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:46.909094    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:47.475677 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:47.475648    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:49.034918 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.034874    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:49.867454 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.867414    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"]
Sep 11 16:22:49.867772 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.867757    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3aff9c12-7bba-41b8-a7ec-839675a54458" containerName="maas-api"
Sep 11 16:22:49.867772 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.867770    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3aff9c12-7bba-41b8-a7ec-839675a54458" containerName="maas-api"
Sep 11 16:22:49.867924 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.867830    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3aff9c12-7bba-41b8-a7ec-839675a54458" containerName="maas-api"
Sep 11 16:22:49.870728 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.870704    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:49.874014 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.873994    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f9d303-serving-cert\""
Sep 11 16:22:49.892261 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.892214    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"]
Sep 11 16:22:49.898494 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.898469    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-metrics-certs\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:49.898620 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.898511    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a053cc-4604-413d-abcb-2041d920ec2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:49.898620 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.898555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:49.898620 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.898612    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjl6\" (UniqueName: \"kubernetes.io/projected/84a053cc-4604-413d-abcb-2041d920ec2b-kube-api-access-nfjl6\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:49.955401 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.955375    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:49.999656 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.999630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjl6\" (UniqueName: \"kubernetes.io/projected/84a053cc-4604-413d-abcb-2041d920ec2b-kube-api-access-nfjl6\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:49.999770 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.999715    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-metrics-certs\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:49.999770 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.999752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a053cc-4604-413d-abcb-2041d920ec2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:49.999883 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:49.999795    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:49.999941 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:22:49.999902    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-serving-cert: secret "maas-api-e2e-ipp-a-f9d303-serving-cert" not found
Sep 11 16:22:49.999997 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:22:49.999966    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls podName:84a053cc-4604-413d-abcb-2041d920ec2b nodeName:}" failed. No retries permitted until 2026-09-11 16:22:50.499944136 +0000 UTC m=+1664.217919236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls") pod "maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt" (UID: "84a053cc-4604-413d-abcb-2041d920ec2b") : secret "maas-api-e2e-ipp-a-f9d303-serving-cert" not found
Sep 11 16:22:50.000445 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:50.000426    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a053cc-4604-413d-abcb-2041d920ec2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:50.001958 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:50.001939    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-metrics-certs\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:50.010187 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:50.010167    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjl6\" (UniqueName: \"kubernetes.io/projected/84a053cc-4604-413d-abcb-2041d920ec2b-kube-api-access-nfjl6\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:50.112989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:50.112956    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:50.503582 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:50.503552    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:50.503733 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:22:50.503681    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-serving-cert: secret "maas-api-e2e-ipp-a-f9d303-serving-cert" not found
Sep 11 16:22:50.503781 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:22:50.503740    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls podName:84a053cc-4604-413d-abcb-2041d920ec2b nodeName:}" failed. No retries permitted until 2026-09-11 16:22:51.503726496 +0000 UTC m=+1665.221701594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls") pod "maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt" (UID: "84a053cc-4604-413d-abcb-2041d920ec2b") : secret "maas-api-e2e-ipp-a-f9d303-serving-cert" not found
Sep 11 16:22:50.565187 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:50.565158    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:50.665930 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:50.665884    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:22:51.512535 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:51.512500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:51.514838 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:51.514813    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:51.681862 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:51.681831    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:51.952827 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:51.952787    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"]
Sep 11 16:22:51.954839 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:22:51.954814    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a053cc_4604_413d_abcb_2041d920ec2b.slice/crio-5bd59308e827e5a22dc502f175b3ec074c8be8ec4fd41983b3fe4e9deecf837d WatchSource:0}: Error finding container 5bd59308e827e5a22dc502f175b3ec074c8be8ec4fd41983b3fe4e9deecf837d: Status 404 returned error can't find the container with id 5bd59308e827e5a22dc502f175b3ec074c8be8ec4fd41983b3fe4e9deecf837d
Sep 11 16:22:52.222304 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:52.222205    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt" event={"ID":"84a053cc-4604-413d-abcb-2041d920ec2b","Type":"ContainerStarted","Data":"5bd59308e827e5a22dc502f175b3ec074c8be8ec4fd41983b3fe4e9deecf837d"}
Sep 11 16:22:53.228043 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:53.228010    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt" event={"ID":"84a053cc-4604-413d-abcb-2041d920ec2b","Type":"ContainerStarted","Data":"a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1"}
Sep 11 16:22:53.228420 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:53.228091    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:22:53.276259 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:53.264871    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt" podStartSLOduration=3.615745087 podStartE2EDuration="4.264850673s" totalImagesPullingTime="649.105586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:22:53.263421814 +0000 UTC m=+1666.981396958" watchObservedRunningTime="2026-09-11 16:22:53.264850673 +0000 UTC m=+1666.982825795"
Sep 11 16:22:57.653680 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.653640    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd" podUID="909c63b7-dcdd-4db2-b9d0-c2a851e561c4" containerName="main" containerID="cri-o://eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a" gracePeriod=45
Sep 11 16:22:57.894779 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.894750    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:22:57.962940 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.962881    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-dshm\") pod \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") "
Sep 11 16:22:57.962940 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.962922    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-model-cache\") pod \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") "
Sep 11 16:22:57.963094 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.962967    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-home\") pod \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") "
Sep 11 16:22:57.963094 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.963020    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tls-certs\") pod \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") "
Sep 11 16:22:57.963094 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.963072    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tmp-dir\") pod \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") "
Sep 11 16:22:57.963258 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.963099    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8frx\" (UniqueName: \"kubernetes.io/projected/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-kube-api-access-t8frx\") pod \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\" (UID: \"909c63b7-dcdd-4db2-b9d0-c2a851e561c4\") "
Sep 11 16:22:57.963258 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.963215    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "909c63b7-dcdd-4db2-b9d0-c2a851e561c4" (UID: "909c63b7-dcdd-4db2-b9d0-c2a851e561c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:22:57.963370 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.963263    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-home" (OuterVolumeSpecName: "home") pod "909c63b7-dcdd-4db2-b9d0-c2a851e561c4" (UID: "909c63b7-dcdd-4db2-b9d0-c2a851e561c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:22:57.963370 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.963354    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "909c63b7-dcdd-4db2-b9d0-c2a851e561c4" (UID: "909c63b7-dcdd-4db2-b9d0-c2a851e561c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:22:57.963477 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.963456    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-home\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:57.963525 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.963478    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tmp-dir\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:57.963525 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.963493    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-model-cache\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:57.965031 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.965009    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-dshm" (OuterVolumeSpecName: "dshm") pod "909c63b7-dcdd-4db2-b9d0-c2a851e561c4" (UID: "909c63b7-dcdd-4db2-b9d0-c2a851e561c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:22:57.965210 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.965189    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-kube-api-access-t8frx" (OuterVolumeSpecName: "kube-api-access-t8frx") pod "909c63b7-dcdd-4db2-b9d0-c2a851e561c4" (UID: "909c63b7-dcdd-4db2-b9d0-c2a851e561c4"). InnerVolumeSpecName "kube-api-access-t8frx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:22:57.965283 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:57.965227    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "909c63b7-dcdd-4db2-b9d0-c2a851e561c4" (UID: "909c63b7-dcdd-4db2-b9d0-c2a851e561c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:22:58.064776 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.064749    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-dshm\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:58.064776 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.064773    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-tls-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:58.064905 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.064787    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8frx\" (UniqueName: \"kubernetes.io/projected/909c63b7-dcdd-4db2-b9d0-c2a851e561c4-kube-api-access-t8frx\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:22:58.250945 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.250873    2564 generic.go:356] "Generic (PLEG): container finished" podID="909c63b7-dcdd-4db2-b9d0-c2a851e561c4" containerID="eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a" exitCode=0
Sep 11 16:22:58.250945 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.250913    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"
Sep 11 16:22:58.251094 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.250959    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd" event={"ID":"909c63b7-dcdd-4db2-b9d0-c2a851e561c4","Type":"ContainerDied","Data":"eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a"}
Sep 11 16:22:58.251094 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.250997    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd" event={"ID":"909c63b7-dcdd-4db2-b9d0-c2a851e561c4","Type":"ContainerDied","Data":"ea77776889f2117cd0ea891fe70f6b68a4d33ff94d519ea40c25daaa2aa2a2c9"}
Sep 11 16:22:58.251094 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.251013    2564 scope.go:117] "RemoveContainer" containerID="eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a"
Sep 11 16:22:58.259285 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.259260    2564 scope.go:117] "RemoveContainer" containerID="eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a"
Sep 11 16:22:58.259549 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:22:58.259529    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a\": container with ID starting with eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a not found: ID does not exist" containerID="eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a"
Sep 11 16:22:58.259604 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.259556    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a"} err="failed to get container status \"eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a\": rpc error: code = NotFound desc = could not find container \"eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a\": container with ID starting with eab5d4bf6b6f2bafa5a1519da782ed8620553cb97dbbad10a34b701fbef76b9a not found: ID does not exist"
Sep 11 16:22:58.275366 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.275339    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"]
Sep 11 16:22:58.287903 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.287877    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-1e0e69/rate-test-model-1e0e69-kserve-f6b56d59f-gmtvd"]
Sep 11 16:22:58.810623 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:58.810589    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="909c63b7-dcdd-4db2-b9d0-c2a851e561c4" path="/var/lib/kubelet/pods/909c63b7-dcdd-4db2-b9d0-c2a851e561c4/volumes"
Sep 11 16:22:59.238417 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:22:59.238344    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:23:12.074695 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:12.074650    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:13.558388 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:13.558352    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:14.758660 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:14.758628    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:15.557976 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:15.557939    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:16.326581 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:16.326547    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:16.743922 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:16.743844    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:17.475096 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:17.475054    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:17.832638 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:17.832600    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:18.171555 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:18.171462    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:20.408639 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:20.408605    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:20.971419 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:20.971387    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:21.082777 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:21.082749    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:23.263601 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:23.263563    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:23.406403 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:23.406370    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:24.485536 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:24.485498    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:25.762270 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.762227    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"]
Sep 11 16:23:25.762633 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.762585    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="909c63b7-dcdd-4db2-b9d0-c2a851e561c4" containerName="main"
Sep 11 16:23:25.762633 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.762596    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="909c63b7-dcdd-4db2-b9d0-c2a851e561c4" containerName="main"
Sep 11 16:23:25.762711 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.762678    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="909c63b7-dcdd-4db2-b9d0-c2a851e561c4" containerName="main"
Sep 11 16:23:25.765801 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.765785    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.772414 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.772393    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c0aa4c-serving-cert\""
Sep 11 16:23:25.814018 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.813994    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"]
Sep 11 16:23:25.866724 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.866703    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9f9257-0325-46d8-b7b1-859c12377a0d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.866833 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.866757    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.866833 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.866816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxc6f\" (UniqueName: \"kubernetes.io/projected/da9f9257-0325-46d8-b7b1-859c12377a0d-kube-api-access-dxc6f\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.866906 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.866856    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.967186 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.967155    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9f9257-0325-46d8-b7b1-859c12377a0d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.967381 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.967284    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.967381 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.967326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxc6f\" (UniqueName: \"kubernetes.io/projected/da9f9257-0325-46d8-b7b1-859c12377a0d-kube-api-access-dxc6f\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.967381 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.967362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.967529 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.967398    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:25.967529 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:23:25.967494    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-serving-cert: secret "maas-api-e2e-ipp-b-c0aa4c-serving-cert" not found
Sep 11 16:23:25.967601 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:23:25.967558    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls podName:da9f9257-0325-46d8-b7b1-859c12377a0d nodeName:}" failed. No retries permitted until 2026-09-11 16:23:26.467539364 +0000 UTC m=+1700.185514466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls") pod "maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5" (UID: "da9f9257-0325-46d8-b7b1-859c12377a0d") : secret "maas-api-e2e-ipp-b-c0aa4c-serving-cert" not found
Sep 11 16:23:25.968033 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.968015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9f9257-0325-46d8-b7b1-859c12377a0d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.969818 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.969799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-metrics-certs\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:25.984084 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:25.984066    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxc6f\" (UniqueName: \"kubernetes.io/projected/da9f9257-0325-46d8-b7b1-859c12377a0d-kube-api-access-dxc6f\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:26.136402 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:26.136366    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:26.471790 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:26.471715    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:26.471925 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:23:26.471859    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-serving-cert: secret "maas-api-e2e-ipp-b-c0aa4c-serving-cert" not found
Sep 11 16:23:26.471925 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:23:26.471916    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls podName:da9f9257-0325-46d8-b7b1-859c12377a0d nodeName:}" failed. No retries permitted until 2026-09-11 16:23:27.471903172 +0000 UTC m=+1701.189878270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls") pod "maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5" (UID: "da9f9257-0325-46d8-b7b1-859c12377a0d") : secret "maas-api-e2e-ipp-b-c0aa4c-serving-cert" not found
Sep 11 16:23:26.768023 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:26.767939    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:26.935622 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:26.935590    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:27.480621 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:27.480592    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:27.483046 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:27.483015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:27.575032 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:27.574998    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:27.869602 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:27.869574    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"]
Sep 11 16:23:27.872998 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:23:27.872963    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9f9257_0325_46d8_b7b1_859c12377a0d.slice/crio-13347e79e9e62824625d080cc97e090de032ddda18c29691e8f30d49594e7d45 WatchSource:0}: Error finding container 13347e79e9e62824625d080cc97e090de032ddda18c29691e8f30d49594e7d45: Status 404 returned error can't find the container with id 13347e79e9e62824625d080cc97e090de032ddda18c29691e8f30d49594e7d45
Sep 11 16:23:28.393376 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:28.393339    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5" event={"ID":"da9f9257-0325-46d8-b7b1-859c12377a0d","Type":"ContainerStarted","Data":"13347e79e9e62824625d080cc97e090de032ddda18c29691e8f30d49594e7d45"}
Sep 11 16:23:29.404771 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:29.404734    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5" event={"ID":"da9f9257-0325-46d8-b7b1-859c12377a0d","Type":"ContainerStarted","Data":"4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720"}
Sep 11 16:23:29.405139 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:29.404912    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:29.436197 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:29.436157    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5" podStartSLOduration=3.402935626 podStartE2EDuration="4.436143626s" totalImagesPullingTime="1.033208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:23:29.435040733 +0000 UTC m=+1703.153015872" watchObservedRunningTime="2026-09-11 16:23:29.436143626 +0000 UTC m=+1703.154118746"
Sep 11 16:23:35.414497 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:35.414466    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:23:49.258523 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:49.258488    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:50.388692 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:50.388654    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:51.141109 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:51.141076    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:51.444370 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:51.444277    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:52.937714 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:52.937679    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:53.921333 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:53.921292    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:23:56.328565 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:23:56.328526    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8slbf_66c21252-78b5-43ab-b332-3922c4d50b79/payload-processing/0.log"
Sep 11 16:24:19.475779 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:19.475749    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:42.713357 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:42.713319    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8slbf_66c21252-78b5-43ab-b332-3922c4d50b79/payload-processing/0.log"
Sep 11 16:24:43.191329 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:43.191293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:43.613576 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:43.613534    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:43.964621 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:43.964543    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:45.800132 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:45.800102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:46.537845 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:46.537814    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:46.747452 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:46.747423    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:47.127052 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:47.127019    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:48.567457 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:48.567421    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:49.722498 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:49.722467    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:50.698893 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:50.698848    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:51.772069 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:51.772029    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"]
Sep 11 16:24:51.775596 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:51.775566    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:51.782158 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:51.781850    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-214daa-serving-cert\""
Sep 11 16:24:51.808179 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:51.808146    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"]
Sep 11 16:24:51.924450 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:51.924422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:51.937417 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:51.937389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5stn\" (UniqueName: \"kubernetes.io/projected/a9fc1a39-4227-430f-932b-b80b11fc00ea-kube-api-access-n5stn\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:51.937548 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:51.937419    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc1a39-4227-430f-932b-b80b11fc00ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:51.937548 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:51.937458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:51.937667 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:51.937603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.019904 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.019875    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:52.037973 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.037914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5stn\" (UniqueName: \"kubernetes.io/projected/a9fc1a39-4227-430f-932b-b80b11fc00ea-kube-api-access-n5stn\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.037973 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.037941    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc1a39-4227-430f-932b-b80b11fc00ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.037973 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.037969    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.038191 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.038035    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.038191 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:24:52.038146    2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-serving-cert: secret "maas-api-e2e-ipp-cleanup-214daa-serving-cert" not found
Sep 11 16:24:52.038340 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:24:52.038210    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-maas-api-tls podName:a9fc1a39-4227-430f-932b-b80b11fc00ea nodeName:}" failed. No retries permitted until 2026-09-11 16:24:52.53819027 +0000 UTC m=+1786.256165588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq" (UID: "a9fc1a39-4227-430f-932b-b80b11fc00ea") : secret "maas-api-e2e-ipp-cleanup-214daa-serving-cert" not found
Sep 11 16:24:52.038701 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.038681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc1a39-4227-430f-932b-b80b11fc00ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.040471 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.040453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.048078 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.048059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5stn\" (UniqueName: \"kubernetes.io/projected/a9fc1a39-4227-430f-932b-b80b11fc00ea-kube-api-access-n5stn\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.457188 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.457151    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:52.542845 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.542816    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.545017 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.544995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.560072 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.560048    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:24:52.685856 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.685831    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:52.710641 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.710566    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"]
Sep 11 16:24:52.714397 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.714378    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:24:52.717444 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.717424    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-214daa\""
Sep 11 16:24:52.717552 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.717533    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-214daa-dockercfg-f8w4p\""
Sep 11 16:24:52.724822 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.724791    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"]
Sep 11 16:24:52.845059 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.845027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd1a760-34f2-4185-8159-ce4870464955-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb\" (UID: \"4cd1a760-34f2-4185-8159-ce4870464955\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:24:52.845448 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.845151    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtp7f\" (UniqueName: \"kubernetes.io/projected/4cd1a760-34f2-4185-8159-ce4870464955-kube-api-access-rtp7f\") pod \"payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb\" (UID: \"4cd1a760-34f2-4185-8159-ce4870464955\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:24:52.946429 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.946401    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd1a760-34f2-4185-8159-ce4870464955-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb\" (UID: \"4cd1a760-34f2-4185-8159-ce4870464955\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:24:52.946577 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.946532    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtp7f\" (UniqueName: \"kubernetes.io/projected/4cd1a760-34f2-4185-8159-ce4870464955-kube-api-access-rtp7f\") pod \"payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb\" (UID: \"4cd1a760-34f2-4185-8159-ce4870464955\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:24:52.947278 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.947236    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd1a760-34f2-4185-8159-ce4870464955-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb\" (UID: \"4cd1a760-34f2-4185-8159-ce4870464955\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:24:52.955914 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.955893    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtp7f\" (UniqueName: \"kubernetes.io/projected/4cd1a760-34f2-4185-8159-ce4870464955-kube-api-access-rtp7f\") pod \"payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb\" (UID: \"4cd1a760-34f2-4185-8159-ce4870464955\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:24:52.969667 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:52.969647    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"]
Sep 11 16:24:52.971841 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:24:52.971816    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fc1a39_4227_430f_932b_b80b11fc00ea.slice/crio-0aa60f488ba1007a4168299488a8ac0f34eecc9709bfd3e283024f11678b2bdf WatchSource:0}: Error finding container 0aa60f488ba1007a4168299488a8ac0f34eecc9709bfd3e283024f11678b2bdf: Status 404 returned error can't find the container with id 0aa60f488ba1007a4168299488a8ac0f34eecc9709bfd3e283024f11678b2bdf
Sep 11 16:24:53.026141 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.026118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:24:53.302582 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.302452    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"]
Sep 11 16:24:53.305787 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:24:53.305752    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd1a760_34f2_4185_8159_ce4870464955.slice/crio-39030884a9efea23e2fcbab499e7f54fcc591d925753a97cc00a291c410db6e2 WatchSource:0}: Error finding container 39030884a9efea23e2fcbab499e7f54fcc591d925753a97cc00a291c410db6e2: Status 404 returned error can't find the container with id 39030884a9efea23e2fcbab499e7f54fcc591d925753a97cc00a291c410db6e2
Sep 11 16:24:53.803056 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.803023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb" event={"ID":"4cd1a760-34f2-4185-8159-ce4870464955","Type":"ContainerStarted","Data":"a825efb6bff1bcc3fd549d4d50bf6e7d3a0742649ab6dbcfbb95c02bdbefe9fd"}
Sep 11 16:24:53.803234 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.803078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb" event={"ID":"4cd1a760-34f2-4185-8159-ce4870464955","Type":"ContainerStarted","Data":"39030884a9efea23e2fcbab499e7f54fcc591d925753a97cc00a291c410db6e2"}
Sep 11 16:24:53.803234 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.803141    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:24:53.804823 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.804792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq" event={"ID":"a9fc1a39-4227-430f-932b-b80b11fc00ea","Type":"ContainerStarted","Data":"0043fb770404bb107354d60e1954c90637c72f75112d216381bc80c94b96921e"}
Sep 11 16:24:53.804926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.804830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq" event={"ID":"a9fc1a39-4227-430f-932b-b80b11fc00ea","Type":"ContainerStarted","Data":"0aa60f488ba1007a4168299488a8ac0f34eecc9709bfd3e283024f11678b2bdf"}
Sep 11 16:24:53.804926 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.804876    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:24:53.833415 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.833372    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb" podStartSLOduration=1.83335957 podStartE2EDuration="1.83335957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:24:53.831175275 +0000 UTC m=+1787.549150394" watchObservedRunningTime="2026-09-11 16:24:53.83335957 +0000 UTC m=+1787.551334689"
Sep 11 16:24:53.875455 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:53.875418    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq" podStartSLOduration=2.42649677 podStartE2EDuration="2.875408453s" totalImagesPullingTime="448.911683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:24:53.874935326 +0000 UTC m=+1787.592910468" watchObservedRunningTime="2026-09-11 16:24:53.875408453 +0000 UTC m=+1787.593383632"
Sep 11 16:24:59.814348 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:24:59.814276    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:25:04.811744 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:04.811713    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:25:14.310235 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:14.310197    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:15.314107 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:15.314071    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:16.479751 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:16.479718    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:16.725992 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:16.725954    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:17.955664 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:17.955630    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:19.099706 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:19.099674    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:20.264576 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:20.264542    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:21.385527 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:21.385489    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:22.143595 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:22.143534    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:22.920445 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:22.920415    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:24.094844 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:24.094809    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:27.966329 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:27.966293    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:28.700131 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:28.700102    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"]
Sep 11 16:25:28.700430 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:28.700383    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq" podUID="a9fc1a39-4227-430f-932b-b80b11fc00ea" containerName="maas-api" containerID="cri-o://0043fb770404bb107354d60e1954c90637c72f75112d216381bc80c94b96921e" gracePeriod=30
Sep 11 16:25:28.789535 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:28.789502    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"]
Sep 11 16:25:28.789727 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:28.789700    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb" podUID="4cd1a760-34f2-4185-8159-ce4870464955" containerName="payload-processing" containerID="cri-o://a825efb6bff1bcc3fd549d4d50bf6e7d3a0742649ab6dbcfbb95c02bdbefe9fd" gracePeriod=30
Sep 11 16:25:28.968703 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:28.968673    2564 generic.go:356] "Generic (PLEG): container finished" podID="4cd1a760-34f2-4185-8159-ce4870464955" containerID="a825efb6bff1bcc3fd549d4d50bf6e7d3a0742649ab6dbcfbb95c02bdbefe9fd" exitCode=0
Sep 11 16:25:28.969015 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:28.968747    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb" event={"ID":"4cd1a760-34f2-4185-8159-ce4870464955","Type":"ContainerDied","Data":"a825efb6bff1bcc3fd549d4d50bf6e7d3a0742649ab6dbcfbb95c02bdbefe9fd"}
Sep 11 16:25:28.970689 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:28.970664    2564 generic.go:356] "Generic (PLEG): container finished" podID="a9fc1a39-4227-430f-932b-b80b11fc00ea" containerID="0043fb770404bb107354d60e1954c90637c72f75112d216381bc80c94b96921e" exitCode=0
Sep 11 16:25:28.970796 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:28.970726    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq" event={"ID":"a9fc1a39-4227-430f-932b-b80b11fc00ea","Type":"ContainerDied","Data":"0043fb770404bb107354d60e1954c90637c72f75112d216381bc80c94b96921e"}
Sep 11 16:25:29.039299 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.039273    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:25:29.067041 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.067019    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:25:29.130264 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.130219    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc1a39-4227-430f-932b-b80b11fc00ea-openshift-service-ca\") pod \"a9fc1a39-4227-430f-932b-b80b11fc00ea\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") "
Sep 11 16:25:29.130372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.130309    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5stn\" (UniqueName: \"kubernetes.io/projected/a9fc1a39-4227-430f-932b-b80b11fc00ea-kube-api-access-n5stn\") pod \"a9fc1a39-4227-430f-932b-b80b11fc00ea\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") "
Sep 11 16:25:29.130372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.130335    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtp7f\" (UniqueName: \"kubernetes.io/projected/4cd1a760-34f2-4185-8159-ce4870464955-kube-api-access-rtp7f\") pod \"4cd1a760-34f2-4185-8159-ce4870464955\" (UID: \"4cd1a760-34f2-4185-8159-ce4870464955\") "
Sep 11 16:25:29.130372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.130361    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-metrics-certs\") pod \"a9fc1a39-4227-430f-932b-b80b11fc00ea\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") "
Sep 11 16:25:29.130546 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.130505    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd1a760-34f2-4185-8159-ce4870464955-plugins-config-volume\") pod \"4cd1a760-34f2-4185-8159-ce4870464955\" (UID: \"4cd1a760-34f2-4185-8159-ce4870464955\") "
Sep 11 16:25:29.130546 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.130528    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9fc1a39-4227-430f-932b-b80b11fc00ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9fc1a39-4227-430f-932b-b80b11fc00ea" (UID: "a9fc1a39-4227-430f-932b-b80b11fc00ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:29.130654 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.130556    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-maas-api-tls\") pod \"a9fc1a39-4227-430f-932b-b80b11fc00ea\" (UID: \"a9fc1a39-4227-430f-932b-b80b11fc00ea\") "
Sep 11 16:25:29.130821 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.130807    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9fc1a39-4227-430f-932b-b80b11fc00ea-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:29.130936 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.130911    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd1a760-34f2-4185-8159-ce4870464955-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4cd1a760-34f2-4185-8159-ce4870464955" (UID: "4cd1a760-34f2-4185-8159-ce4870464955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:29.132438 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.132416    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd1a760-34f2-4185-8159-ce4870464955-kube-api-access-rtp7f" (OuterVolumeSpecName: "kube-api-access-rtp7f") pod "4cd1a760-34f2-4185-8159-ce4870464955" (UID: "4cd1a760-34f2-4185-8159-ce4870464955"). InnerVolumeSpecName "kube-api-access-rtp7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:25:29.132531 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.132499    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9fc1a39-4227-430f-932b-b80b11fc00ea" (UID: "a9fc1a39-4227-430f-932b-b80b11fc00ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:25:29.132794 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.132769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a9fc1a39-4227-430f-932b-b80b11fc00ea" (UID: "a9fc1a39-4227-430f-932b-b80b11fc00ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:25:29.132794 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.132775    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9fc1a39-4227-430f-932b-b80b11fc00ea-kube-api-access-n5stn" (OuterVolumeSpecName: "kube-api-access-n5stn") pod "a9fc1a39-4227-430f-932b-b80b11fc00ea" (UID: "a9fc1a39-4227-430f-932b-b80b11fc00ea"). InnerVolumeSpecName "kube-api-access-n5stn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:25:29.231461 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.231399    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:29.231461 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.231433    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5stn\" (UniqueName: \"kubernetes.io/projected/a9fc1a39-4227-430f-932b-b80b11fc00ea-kube-api-access-n5stn\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:29.231461 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.231443    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtp7f\" (UniqueName: \"kubernetes.io/projected/4cd1a760-34f2-4185-8159-ce4870464955-kube-api-access-rtp7f\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:29.231461 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.231452    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9fc1a39-4227-430f-932b-b80b11fc00ea-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:29.231461 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.231461    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cd1a760-34f2-4185-8159-ce4870464955-plugins-config-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:29.706646 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.706607    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:29.976178 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.976102    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"
Sep 11 16:25:29.976178 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.976108    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb" event={"ID":"4cd1a760-34f2-4185-8159-ce4870464955","Type":"ContainerDied","Data":"39030884a9efea23e2fcbab499e7f54fcc591d925753a97cc00a291c410db6e2"}
Sep 11 16:25:29.976178 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.976173    2564 scope.go:117] "RemoveContainer" containerID="a825efb6bff1bcc3fd549d4d50bf6e7d3a0742649ab6dbcfbb95c02bdbefe9fd"
Sep 11 16:25:29.977536 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.977515    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"
Sep 11 16:25:29.977536 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.977526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq" event={"ID":"a9fc1a39-4227-430f-932b-b80b11fc00ea","Type":"ContainerDied","Data":"0aa60f488ba1007a4168299488a8ac0f34eecc9709bfd3e283024f11678b2bdf"}
Sep 11 16:25:29.985145 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:29.985128    2564 scope.go:117] "RemoveContainer" containerID="0043fb770404bb107354d60e1954c90637c72f75112d216381bc80c94b96921e"
Sep 11 16:25:30.006976 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:30.006950    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"]
Sep 11 16:25:30.010953 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:30.010934    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-214daa-656dc46d54-trdbq"]
Sep 11 16:25:30.039154 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:30.039128    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"]
Sep 11 16:25:30.044997 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:30.044976    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-214daa-57f48864f-f9bqb"]
Sep 11 16:25:30.497538 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:30.497505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:30.812807 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:30.812773    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cd1a760-34f2-4185-8159-ce4870464955" path="/var/lib/kubelet/pods/4cd1a760-34f2-4185-8159-ce4870464955/volumes"
Sep 11 16:25:30.813253 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:30.813223    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9fc1a39-4227-430f-932b-b80b11fc00ea" path="/var/lib/kubelet/pods/a9fc1a39-4227-430f-932b-b80b11fc00ea/volumes"
Sep 11 16:25:31.298818 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:31.298749    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:32.286948 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:32.286916    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:33.107844 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:33.107796    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:35.538782 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:35.538745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:35.965224 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:35.965193    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:36.037033 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:36.037006    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:39.859227 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:39.859192    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:40.765819 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:40.765781    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:41.749274 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:41.749222    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:42.746946 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:42.746911    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:43.611405 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:43.611359    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:44.530929 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:44.530895    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:46.489844 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:46.489801    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:55.295178 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.295144    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"]
Sep 11 16:25:55.295640 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.295335    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt" podUID="84a053cc-4604-413d-abcb-2041d920ec2b" containerName="maas-api" containerID="cri-o://a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1" gracePeriod=30
Sep 11 16:25:55.352129 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.352091    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:55.498359 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.498327    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:55.556661 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.556600    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:25:55.604108 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.604084    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:55.637889 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.637867    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls\") pod \"84a053cc-4604-413d-abcb-2041d920ec2b\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") "
Sep 11 16:25:55.638004 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.637933    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a053cc-4604-413d-abcb-2041d920ec2b-openshift-service-ca\") pod \"84a053cc-4604-413d-abcb-2041d920ec2b\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") "
Sep 11 16:25:55.638102 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.638085    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfjl6\" (UniqueName: \"kubernetes.io/projected/84a053cc-4604-413d-abcb-2041d920ec2b-kube-api-access-nfjl6\") pod \"84a053cc-4604-413d-abcb-2041d920ec2b\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") "
Sep 11 16:25:55.638201 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.638189    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-metrics-certs\") pod \"84a053cc-4604-413d-abcb-2041d920ec2b\" (UID: \"84a053cc-4604-413d-abcb-2041d920ec2b\") "
Sep 11 16:25:55.638282 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.638266    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a053cc-4604-413d-abcb-2041d920ec2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "84a053cc-4604-413d-abcb-2041d920ec2b" (UID: "84a053cc-4604-413d-abcb-2041d920ec2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:25:55.638477 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.638453    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84a053cc-4604-413d-abcb-2041d920ec2b-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:55.639992 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.639967    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "84a053cc-4604-413d-abcb-2041d920ec2b" (UID: "84a053cc-4604-413d-abcb-2041d920ec2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:25:55.640128 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.640107    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a053cc-4604-413d-abcb-2041d920ec2b-kube-api-access-nfjl6" (OuterVolumeSpecName: "kube-api-access-nfjl6") pod "84a053cc-4604-413d-abcb-2041d920ec2b" (UID: "84a053cc-4604-413d-abcb-2041d920ec2b"). InnerVolumeSpecName "kube-api-access-nfjl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:25:55.640206 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.640182    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "84a053cc-4604-413d-abcb-2041d920ec2b" (UID: "84a053cc-4604-413d-abcb-2041d920ec2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:25:55.704125 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.704102    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:55.739356 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.739332    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjl6\" (UniqueName: \"kubernetes.io/projected/84a053cc-4604-413d-abcb-2041d920ec2b-kube-api-access-nfjl6\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:55.739356 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.739353    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:55.739477 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.739362    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84a053cc-4604-413d-abcb-2041d920ec2b-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:25:55.778997 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:55.778378    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:56.111795 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.111762    2564 generic.go:356] "Generic (PLEG): container finished" podID="84a053cc-4604-413d-abcb-2041d920ec2b" containerID="a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1" exitCode=0
Sep 11 16:25:56.111967 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.111802    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"
Sep 11 16:25:56.111967 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.111837    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt" event={"ID":"84a053cc-4604-413d-abcb-2041d920ec2b","Type":"ContainerDied","Data":"a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1"}
Sep 11 16:25:56.111967 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.111889    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt" event={"ID":"84a053cc-4604-413d-abcb-2041d920ec2b","Type":"ContainerDied","Data":"5bd59308e827e5a22dc502f175b3ec074c8be8ec4fd41983b3fe4e9deecf837d"}
Sep 11 16:25:56.111967 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.111911    2564 scope.go:117] "RemoveContainer" containerID="a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1"
Sep 11 16:25:56.120893 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.120878    2564 scope.go:117] "RemoveContainer" containerID="a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1"
Sep 11 16:25:56.121134 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:25:56.121117    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1\": container with ID starting with a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1 not found: ID does not exist" containerID="a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1"
Sep 11 16:25:56.121263 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.121142    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1"} err="failed to get container status \"a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1\": rpc error: code = NotFound desc = could not find container \"a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1\": container with ID starting with a6cbaefd562d44b3cb2aa706889a015cf996cf83cfd2311587be9f026b2757d1 not found: ID does not exist"
Sep 11 16:25:56.135763 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.135739    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"]
Sep 11 16:25:56.141290 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.141266    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f9d303-688b7dc787-k9hlt"]
Sep 11 16:25:56.146121 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.146101    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:56.291988 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.291959    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:56.811379 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:56.811342    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84a053cc-4604-413d-abcb-2041d920ec2b" path="/var/lib/kubelet/pods/84a053cc-4604-413d-abcb-2041d920ec2b/volumes"
Sep 11 16:25:57.691834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:57.691800    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:58.754437 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:58.754396    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:25:59.570456 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:25:59.570426    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:01.149202 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:01.149173    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:03.957017 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:03.956980    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:05.045106 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:05.045070    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:23.431012 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:23.430979    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:37.715650 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:37.715566    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:38.268468 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:38.268434    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:44.045010 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:44.044969    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:44.147042 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:44.147006    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:44.384741 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:44.384706    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:44.947846 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:44.947808    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:45.836492 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:45.836458    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:46.769565 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:46.769531    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:47.535097 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:47.535067    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:48.435697 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:48.435669    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:54.427517 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:54.427478    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:54.528513 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:54.528482    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:55.408455 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.408421    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"]
Sep 11 16:26:55.408715 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.408610    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5" podUID="da9f9257-0325-46d8-b7b1-859c12377a0d" containerName="maas-api" containerID="cri-o://4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720" gracePeriod=30
Sep 11 16:26:55.468902 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.468860    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:55.549996 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.549964    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:55.650529 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.650500    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:55.662254 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.662188    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:26:55.732423 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.732399    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:55.793020 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.792992    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls\") pod \"da9f9257-0325-46d8-b7b1-859c12377a0d\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") "
Sep 11 16:26:55.793154 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.793093    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9f9257-0325-46d8-b7b1-859c12377a0d-openshift-service-ca\") pod \"da9f9257-0325-46d8-b7b1-859c12377a0d\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") "
Sep 11 16:26:55.793154 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.793121    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxc6f\" (UniqueName: \"kubernetes.io/projected/da9f9257-0325-46d8-b7b1-859c12377a0d-kube-api-access-dxc6f\") pod \"da9f9257-0325-46d8-b7b1-859c12377a0d\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") "
Sep 11 16:26:55.793293 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.793154    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-metrics-certs\") pod \"da9f9257-0325-46d8-b7b1-859c12377a0d\" (UID: \"da9f9257-0325-46d8-b7b1-859c12377a0d\") "
Sep 11 16:26:55.793543 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.793513    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9f9257-0325-46d8-b7b1-859c12377a0d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da9f9257-0325-46d8-b7b1-859c12377a0d" (UID: "da9f9257-0325-46d8-b7b1-859c12377a0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:26:55.795149 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.795128    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da9f9257-0325-46d8-b7b1-859c12377a0d" (UID: "da9f9257-0325-46d8-b7b1-859c12377a0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:26:55.795278 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.795153    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "da9f9257-0325-46d8-b7b1-859c12377a0d" (UID: "da9f9257-0325-46d8-b7b1-859c12377a0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:26:55.795333 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.795288    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9f9257-0325-46d8-b7b1-859c12377a0d-kube-api-access-dxc6f" (OuterVolumeSpecName: "kube-api-access-dxc6f") pod "da9f9257-0325-46d8-b7b1-859c12377a0d" (UID: "da9f9257-0325-46d8-b7b1-859c12377a0d"). InnerVolumeSpecName "kube-api-access-dxc6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:26:55.894065 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.894027    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da9f9257-0325-46d8-b7b1-859c12377a0d-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:26:55.894065 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.894056    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxc6f\" (UniqueName: \"kubernetes.io/projected/da9f9257-0325-46d8-b7b1-859c12377a0d-kube-api-access-dxc6f\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:26:55.894065 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.894067    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:26:55.894065 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:55.894075    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da9f9257-0325-46d8-b7b1-859c12377a0d-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:26:56.167812 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.167779    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:56.296138 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.296103    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:56.397114 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.397088    2564 generic.go:356] "Generic (PLEG): container finished" podID="da9f9257-0325-46d8-b7b1-859c12377a0d" containerID="4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720" exitCode=0
Sep 11 16:26:56.397294 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.397137    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"
Sep 11 16:26:56.397294 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.397177    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5" event={"ID":"da9f9257-0325-46d8-b7b1-859c12377a0d","Type":"ContainerDied","Data":"4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720"}
Sep 11 16:26:56.397294 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.397224    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5" event={"ID":"da9f9257-0325-46d8-b7b1-859c12377a0d","Type":"ContainerDied","Data":"13347e79e9e62824625d080cc97e090de032ddda18c29691e8f30d49594e7d45"}
Sep 11 16:26:56.397294 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.397259    2564 scope.go:117] "RemoveContainer" containerID="4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720"
Sep 11 16:26:56.406158 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.406142    2564 scope.go:117] "RemoveContainer" containerID="4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720"
Sep 11 16:26:56.406454 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:26:56.406433    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720\": container with ID starting with 4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720 not found: ID does not exist" containerID="4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720"
Sep 11 16:26:56.406537 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.406460    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720"} err="failed to get container status \"4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720\": rpc error: code = NotFound desc = could not find container \"4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720\": container with ID starting with 4d23efe2515f03b70badb50589aa178ba2c8395acba877a5a2a742400b11c720 not found: ID does not exist"
Sep 11 16:26:56.432922 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.432872    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"]
Sep 11 16:26:56.435451 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.435434    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c0aa4c-6cc7cf5fbf-mhzb5"]
Sep 11 16:26:56.811104 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:56.811078    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da9f9257-0325-46d8-b7b1-859c12377a0d" path="/var/lib/kubelet/pods/da9f9257-0325-46d8-b7b1-859c12377a0d/volumes"
Sep 11 16:26:57.105302 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:57.105204    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:26:57.844752 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:26:57.844719    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:02.279349 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:02.279314    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:02.379919 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:02.379882    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:02.598447 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:02.598422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:03.383283 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:03.383229    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:04.561953 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:04.561923    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:05.409297 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:05.409263    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:06.144865 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:06.144837    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:13.655734 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:13.655703    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"]
Sep 11 16:27:13.656154 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:13.655913    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9" podUID="c85eb56c-fc76-4349-b115-1c5f50046a7c" containerName="maas-api" containerID="cri-o://b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395" gracePeriod=30
Sep 11 16:27:13.687277 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:13.687226    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:13.788502 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:13.788466    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:13.887561 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:13.887537    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:13.894988 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:13.894967    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:27:13.987814 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:13.987740    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:14.038357 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.038327    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6g8t\" (UniqueName: \"kubernetes.io/projected/c85eb56c-fc76-4349-b115-1c5f50046a7c-kube-api-access-h6g8t\") pod \"c85eb56c-fc76-4349-b115-1c5f50046a7c\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") "
Sep 11 16:27:14.038521 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.038366    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-maas-api-tls\") pod \"c85eb56c-fc76-4349-b115-1c5f50046a7c\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") "
Sep 11 16:27:14.038587 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.038528    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85eb56c-fc76-4349-b115-1c5f50046a7c-openshift-service-ca\") pod \"c85eb56c-fc76-4349-b115-1c5f50046a7c\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") "
Sep 11 16:27:14.038587 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.038562    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-metrics-certs\") pod \"c85eb56c-fc76-4349-b115-1c5f50046a7c\" (UID: \"c85eb56c-fc76-4349-b115-1c5f50046a7c\") "
Sep 11 16:27:14.038836 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.038810    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85eb56c-fc76-4349-b115-1c5f50046a7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c85eb56c-fc76-4349-b115-1c5f50046a7c" (UID: "c85eb56c-fc76-4349-b115-1c5f50046a7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:14.040588 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.040544    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c85eb56c-fc76-4349-b115-1c5f50046a7c" (UID: "c85eb56c-fc76-4349-b115-1c5f50046a7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:14.040682 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.040622    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85eb56c-fc76-4349-b115-1c5f50046a7c-kube-api-access-h6g8t" (OuterVolumeSpecName: "kube-api-access-h6g8t") pod "c85eb56c-fc76-4349-b115-1c5f50046a7c" (UID: "c85eb56c-fc76-4349-b115-1c5f50046a7c"). InnerVolumeSpecName "kube-api-access-h6g8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:14.040903 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.040879    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c85eb56c-fc76-4349-b115-1c5f50046a7c" (UID: "c85eb56c-fc76-4349-b115-1c5f50046a7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:14.139424 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.139391    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c85eb56c-fc76-4349-b115-1c5f50046a7c-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.139424 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.139418    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.139424 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.139427    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6g8t\" (UniqueName: \"kubernetes.io/projected/c85eb56c-fc76-4349-b115-1c5f50046a7c-kube-api-access-h6g8t\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.139612 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.139436    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c85eb56c-fc76-4349-b115-1c5f50046a7c-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:14.340016 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.339985    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:14.451340 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.451313    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:14.485593 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.485571    2564 generic.go:356] "Generic (PLEG): container finished" podID="c85eb56c-fc76-4349-b115-1c5f50046a7c" containerID="b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395" exitCode=0
Sep 11 16:27:14.485716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.485609    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9" event={"ID":"c85eb56c-fc76-4349-b115-1c5f50046a7c","Type":"ContainerDied","Data":"b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395"}
Sep 11 16:27:14.485716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.485675    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9" event={"ID":"c85eb56c-fc76-4349-b115-1c5f50046a7c","Type":"ContainerDied","Data":"3b55cd189b8a0f3f775803853d93051e1017150daf7eff85eb208f54de9e5f61"}
Sep 11 16:27:14.485716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.485689    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"
Sep 11 16:27:14.485834 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.485696    2564 scope.go:117] "RemoveContainer" containerID="b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395"
Sep 11 16:27:14.495046 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.495028    2564 scope.go:117] "RemoveContainer" containerID="b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395"
Sep 11 16:27:14.495333 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:27:14.495314    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395\": container with ID starting with b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395 not found: ID does not exist" containerID="b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395"
Sep 11 16:27:14.495423 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.495341    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395"} err="failed to get container status \"b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395\": rpc error: code = NotFound desc = could not find container \"b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395\": container with ID starting with b482da454cf6e26e3bcb5119a8de66aa8bc012bed309d642302b810a69f4a395 not found: ID does not exist"
Sep 11 16:27:14.508325 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.508301    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"]
Sep 11 16:27:14.514096 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.514077    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-59f108-7f6d956768-c44l9"]
Sep 11 16:27:14.810173 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:14.810119    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85eb56c-fc76-4349-b115-1c5f50046a7c" path="/var/lib/kubelet/pods/c85eb56c-fc76-4349-b115-1c5f50046a7c/volumes"
Sep 11 16:27:15.260261 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:15.260171    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:20.913010 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:20.912971    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:20.979705 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:20.979679    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:21.186880 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:21.186805    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:22.549533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:22.549505    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:23.337458 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:23.337422    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.076301 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.076268    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"]
Sep 11 16:27:32.076694 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.076474    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw" podUID="2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" containerName="maas-api" containerID="cri-o://992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a" gracePeriod=30
Sep 11 16:27:32.104813 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.104785    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.124268 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:27:32.124225    2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9f8e2b_6f88_47d1_93cc_89a7a8a97899.slice/crio-conmon-992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 16:27:32.135416 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.135390    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"]
Sep 11 16:27:32.135631 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.135606    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t" podUID="c1ad0afd-ab96-4e44-9abf-a2b307dc6657" containerName="payload-pre-processing" containerID="cri-o://a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36" gracePeriod=30
Sep 11 16:27:32.204843 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.204808    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.304652 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.304597    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.374495 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.374471    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:27:32.377155 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.377129    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.408090 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.408069    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:27:32.471630 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.471597    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-maas-api-tls\") pod \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") "
Sep 11 16:27:32.471781 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.471636    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-metrics-certs\") pod \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") "
Sep 11 16:27:32.471781 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.471711    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csvdv\" (UniqueName: \"kubernetes.io/projected/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-kube-api-access-csvdv\") pod \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") "
Sep 11 16:27:32.471781 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.471750    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-openshift-service-ca\") pod \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\" (UID: \"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899\") "
Sep 11 16:27:32.472290 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.472234    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" (UID: "2f9f8e2b-6f88-47d1-93cc-89a7a8a97899"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:32.473783 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.473750    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" (UID: "2f9f8e2b-6f88-47d1-93cc-89a7a8a97899"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:32.473998 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.473977    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" (UID: "2f9f8e2b-6f88-47d1-93cc-89a7a8a97899"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:32.474215 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.474193    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-kube-api-access-csvdv" (OuterVolumeSpecName: "kube-api-access-csvdv") pod "2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" (UID: "2f9f8e2b-6f88-47d1-93cc-89a7a8a97899"). InnerVolumeSpecName "kube-api-access-csvdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:32.572549 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.572519    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p7hd\" (UniqueName: \"kubernetes.io/projected/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-kube-api-access-5p7hd\") pod \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\" (UID: \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\") "
Sep 11 16:27:32.572688 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.572608    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-plugins-config-volume\") pod \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\" (UID: \"c1ad0afd-ab96-4e44-9abf-a2b307dc6657\") "
Sep 11 16:27:32.572897 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.572880    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:32.572969 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.572903    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:32.572969 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.572918    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csvdv\" (UniqueName: \"kubernetes.io/projected/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-kube-api-access-csvdv\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:32.572969 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.572936    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:32.572969 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.572949    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1ad0afd-ab96-4e44-9abf-a2b307dc6657" (UID: "c1ad0afd-ab96-4e44-9abf-a2b307dc6657"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:27:32.574317 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.574287    2564 generic.go:356] "Generic (PLEG): container finished" podID="c1ad0afd-ab96-4e44-9abf-a2b307dc6657" containerID="a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36" exitCode=0
Sep 11 16:27:32.574464 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.574341    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"
Sep 11 16:27:32.574464 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.574363    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t" event={"ID":"c1ad0afd-ab96-4e44-9abf-a2b307dc6657","Type":"ContainerDied","Data":"a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36"}
Sep 11 16:27:32.574464 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.574427    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t" event={"ID":"c1ad0afd-ab96-4e44-9abf-a2b307dc6657","Type":"ContainerDied","Data":"b6dc325925a1f1c00c8a8b76102433f5995b754358488d4ffe2b324436515a08"}
Sep 11 16:27:32.574464 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.574460    2564 scope.go:117] "RemoveContainer" containerID="a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36"
Sep 11 16:27:32.574798 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.574776    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-kube-api-access-5p7hd" (OuterVolumeSpecName: "kube-api-access-5p7hd") pod "c1ad0afd-ab96-4e44-9abf-a2b307dc6657" (UID: "c1ad0afd-ab96-4e44-9abf-a2b307dc6657"). InnerVolumeSpecName "kube-api-access-5p7hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:32.575909 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.575888    2564 generic.go:356] "Generic (PLEG): container finished" podID="2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" containerID="992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a" exitCode=0
Sep 11 16:27:32.576053 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.575916    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw" event={"ID":"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899","Type":"ContainerDied","Data":"992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a"}
Sep 11 16:27:32.576053 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.575938    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"
Sep 11 16:27:32.576053 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.575964    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw" event={"ID":"2f9f8e2b-6f88-47d1-93cc-89a7a8a97899","Type":"ContainerDied","Data":"bdd73acd6227b9b6cb881e035136af0516e55a465b68f51c151de6f1ffdbbb62"}
Sep 11 16:27:32.583019 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.582976    2564 scope.go:117] "RemoveContainer" containerID="a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36"
Sep 11 16:27:32.583326 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:27:32.583295    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36\": container with ID starting with a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36 not found: ID does not exist" containerID="a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36"
Sep 11 16:27:32.583474 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.583336    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36"} err="failed to get container status \"a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36\": rpc error: code = NotFound desc = could not find container \"a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36\": container with ID starting with a7459cb83587e1c4ff0b69878109afd5a95a6689f9ca0d5e585b6b787e6f6f36 not found: ID does not exist"
Sep 11 16:27:32.583474 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.583357    2564 scope.go:117] "RemoveContainer" containerID="992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a"
Sep 11 16:27:32.590784 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.590766    2564 scope.go:117] "RemoveContainer" containerID="992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a"
Sep 11 16:27:32.591057 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:27:32.591039    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a\": container with ID starting with 992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a not found: ID does not exist" containerID="992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a"
Sep 11 16:27:32.591115 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.591062    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a"} err="failed to get container status \"992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a\": rpc error: code = NotFound desc = could not find container \"992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a\": container with ID starting with 992e2d1f90f3dc8452d03d7a8829de1fe48a64c174ffb994977d7c64b82b184a not found: ID does not exist"
Sep 11 16:27:32.598100 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.598070    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"]
Sep 11 16:27:32.605697 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.605680    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1e0e69-6b9557788b-5x5lw"]
Sep 11 16:27:32.673681 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.673657    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-plugins-config-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:32.673681 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.673676    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p7hd\" (UniqueName: \"kubernetes.io/projected/c1ad0afd-ab96-4e44-9abf-a2b307dc6657-kube-api-access-5p7hd\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:32.718375 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.718357    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.811116 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.811092    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" path="/var/lib/kubelet/pods/2f9f8e2b-6f88-47d1-93cc-89a7a8a97899/volumes"
Sep 11 16:27:32.820492 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.820472    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:32.892086 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.892022    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"]
Sep 11 16:27:32.896369 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:32.896348    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-1e0e69-5b899688d6-nrp7t"]
Sep 11 16:27:34.810670 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:34.810637    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1ad0afd-ab96-4e44-9abf-a2b307dc6657" path="/var/lib/kubelet/pods/c1ad0afd-ab96-4e44-9abf-a2b307dc6657/volumes"
Sep 11 16:27:38.850763 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:38.850728    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:38.953032 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:38.952998    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:39.156763 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:39.156651    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:27:41.708618 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:41.708581    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-fz5g4"]
Sep 11 16:27:41.709438 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:41.709382    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f55cd5c49-fz5g4" podUID="b310a818-e4e7-4463-b814-2ac0f9d6450c" containerName="manager" containerID="cri-o://952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457" gracePeriod=30
Sep 11 16:27:41.950547 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:41.950521    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:27:42.038345 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.038273    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj7hx\" (UniqueName: \"kubernetes.io/projected/b310a818-e4e7-4463-b814-2ac0f9d6450c-kube-api-access-sj7hx\") pod \"b310a818-e4e7-4463-b814-2ac0f9d6450c\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") "
Sep 11 16:27:42.038345 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.038306    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-webhook-cert\") pod \"b310a818-e4e7-4463-b814-2ac0f9d6450c\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") "
Sep 11 16:27:42.038345 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.038328    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-metrics-certs\") pod \"b310a818-e4e7-4463-b814-2ac0f9d6450c\" (UID: \"b310a818-e4e7-4463-b814-2ac0f9d6450c\") "
Sep 11 16:27:42.040265 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.040213    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b310a818-e4e7-4463-b814-2ac0f9d6450c" (UID: "b310a818-e4e7-4463-b814-2ac0f9d6450c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:42.040378 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.040324    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b310a818-e4e7-4463-b814-2ac0f9d6450c-kube-api-access-sj7hx" (OuterVolumeSpecName: "kube-api-access-sj7hx") pod "b310a818-e4e7-4463-b814-2ac0f9d6450c" (UID: "b310a818-e4e7-4463-b814-2ac0f9d6450c"). InnerVolumeSpecName "kube-api-access-sj7hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:27:42.040440 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.040379    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b310a818-e4e7-4463-b814-2ac0f9d6450c" (UID: "b310a818-e4e7-4463-b814-2ac0f9d6450c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:27:42.138818 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.138792    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj7hx\" (UniqueName: \"kubernetes.io/projected/b310a818-e4e7-4463-b814-2ac0f9d6450c-kube-api-access-sj7hx\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:42.138818 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.138815    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-webhook-cert\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:42.138818 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.138824    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b310a818-e4e7-4463-b814-2ac0f9d6450c-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:27:42.625095 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.625061    2564 generic.go:356] "Generic (PLEG): container finished" podID="b310a818-e4e7-4463-b814-2ac0f9d6450c" containerID="952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457" exitCode=0
Sep 11 16:27:42.625298 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.625118    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-fz5g4"
Sep 11 16:27:42.625298 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.625145    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-fz5g4" event={"ID":"b310a818-e4e7-4463-b814-2ac0f9d6450c","Type":"ContainerDied","Data":"952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457"}
Sep 11 16:27:42.625298 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.625217    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-fz5g4" event={"ID":"b310a818-e4e7-4463-b814-2ac0f9d6450c","Type":"ContainerDied","Data":"0b2ddc8c81c38a6e5eed409e1b7baee01940d2f93dadb766af144df4f527dcd0"}
Sep 11 16:27:42.625298 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.625251    2564 scope.go:117] "RemoveContainer" containerID="952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457"
Sep 11 16:27:42.634541 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.634524    2564 scope.go:117] "RemoveContainer" containerID="952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457"
Sep 11 16:27:42.634789 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:27:42.634769    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457\": container with ID starting with 952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457 not found: ID does not exist" containerID="952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457"
Sep 11 16:27:42.634843 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.634797    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457"} err="failed to get container status \"952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457\": rpc error: code = NotFound desc = could not find container \"952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457\": container with ID starting with 952e5625e5a43d0db3f2371cfe7c6567d977576c23f983a142ea0b8756fff457 not found: ID does not exist"
Sep 11 16:27:42.646936 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.646916    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-fz5g4"]
Sep 11 16:27:42.650789 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.650770    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-fz5g4"]
Sep 11 16:27:42.810487 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:42.810462    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b310a818-e4e7-4463-b814-2ac0f9d6450c" path="/var/lib/kubelet/pods/b310a818-e4e7-4463-b814-2ac0f9d6450c/volumes"
Sep 11 16:27:52.667681 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:27:52.667648    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:05.161376 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:05.161282    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:05.221713 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:05.221681    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:15.192220 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:15.192172    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:25.761280 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:25.761222    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:28:43.446222 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446184    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f55cd5c49-6nw59"]
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446556    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b310a818-e4e7-4463-b814-2ac0f9d6450c" containerName="manager"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446572    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b310a818-e4e7-4463-b814-2ac0f9d6450c" containerName="manager"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446580    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1ad0afd-ab96-4e44-9abf-a2b307dc6657" containerName="payload-pre-processing"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446588    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1ad0afd-ab96-4e44-9abf-a2b307dc6657" containerName="payload-pre-processing"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446605    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da9f9257-0325-46d8-b7b1-859c12377a0d" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446611    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="da9f9257-0325-46d8-b7b1-859c12377a0d" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446621    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446626    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446632    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85eb56c-fc76-4349-b115-1c5f50046a7c" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446637    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85eb56c-fc76-4349-b115-1c5f50046a7c" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446644    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84a053cc-4604-413d-abcb-2041d920ec2b" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446649    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="84a053cc-4604-413d-abcb-2041d920ec2b" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446654    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9fc1a39-4227-430f-932b-b80b11fc00ea" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446659    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9fc1a39-4227-430f-932b-b80b11fc00ea" containerName="maas-api"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446669    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cd1a760-34f2-4185-8159-ce4870464955" containerName="payload-processing"
Sep 11 16:28:43.446716 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446674    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cd1a760-34f2-4185-8159-ce4870464955" containerName="payload-processing"
Sep 11 16:28:43.447268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446729    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="da9f9257-0325-46d8-b7b1-859c12377a0d" containerName="maas-api"
Sep 11 16:28:43.447268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446738    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cd1a760-34f2-4185-8159-ce4870464955" containerName="payload-processing"
Sep 11 16:28:43.447268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446745    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b310a818-e4e7-4463-b814-2ac0f9d6450c" containerName="manager"
Sep 11 16:28:43.447268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446751    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9fc1a39-4227-430f-932b-b80b11fc00ea" containerName="maas-api"
Sep 11 16:28:43.447268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446758    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85eb56c-fc76-4349-b115-1c5f50046a7c" containerName="maas-api"
Sep 11 16:28:43.447268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446764    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="84a053cc-4604-413d-abcb-2041d920ec2b" containerName="maas-api"
Sep 11 16:28:43.447268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446769    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9f8e2b-6f88-47d1-93cc-89a7a8a97899" containerName="maas-api"
Sep 11 16:28:43.447268 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.446775    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1ad0afd-ab96-4e44-9abf-a2b307dc6657" containerName="payload-pre-processing"
Sep 11 16:28:43.449910 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.449884    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.454216 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.454186    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w4bfh\""
Sep 11 16:28:43.454350 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.454186    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 16:28:43.454570 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.454549    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 16:28:43.454629 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.454574    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 16:28:43.461332 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.461303    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-6nw59"]
Sep 11 16:28:43.570651 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.570622    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-metrics-certs\") pod \"maas-controller-f55cd5c49-6nw59\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") " pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.570768 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.570667    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-webhook-cert\") pod \"maas-controller-f55cd5c49-6nw59\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") " pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.570825 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.570776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfhk2\" (UniqueName: \"kubernetes.io/projected/78cb444b-65ba-4b70-ba7b-8d0019e2e043-kube-api-access-wfhk2\") pod \"maas-controller-f55cd5c49-6nw59\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") " pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.672112 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.672086    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfhk2\" (UniqueName: \"kubernetes.io/projected/78cb444b-65ba-4b70-ba7b-8d0019e2e043-kube-api-access-wfhk2\") pod \"maas-controller-f55cd5c49-6nw59\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") " pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.672279 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.672124    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-metrics-certs\") pod \"maas-controller-f55cd5c49-6nw59\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") " pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.672279 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.672161    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-webhook-cert\") pod \"maas-controller-f55cd5c49-6nw59\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") " pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.674870 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.674844    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-webhook-cert\") pod \"maas-controller-f55cd5c49-6nw59\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") " pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.675102 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.675077    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-metrics-certs\") pod \"maas-controller-f55cd5c49-6nw59\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") " pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.683212 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.683185    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfhk2\" (UniqueName: \"kubernetes.io/projected/78cb444b-65ba-4b70-ba7b-8d0019e2e043-kube-api-access-wfhk2\") pod \"maas-controller-f55cd5c49-6nw59\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") " pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:43.762166 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:43.762096    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:44.042166 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:44.042137    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-6nw59"]
Sep 11 16:28:44.050924 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:44.050892    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 16:28:44.905708 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:44.905675    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-6nw59" event={"ID":"78cb444b-65ba-4b70-ba7b-8d0019e2e043","Type":"ContainerStarted","Data":"7145928c6e49ddf7569b8135d5294ba22fce167b602354723448f21995d02e12"}
Sep 11 16:28:45.911573 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:45.911536    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-6nw59" event={"ID":"78cb444b-65ba-4b70-ba7b-8d0019e2e043","Type":"ContainerStarted","Data":"b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37"}
Sep 11 16:28:45.912058 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:45.911674    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:28:45.931273 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:45.931209    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f55cd5c49-6nw59" podStartSLOduration=1.965940816 podStartE2EDuration="2.931193248s" totalImagesPullingTime="965.252432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:28:45.929790291 +0000 UTC m=+2019.647765429" watchObservedRunningTime="2026-09-11 16:28:45.931193248 +0000 UTC m=+2019.649168368"
Sep 11 16:28:56.918431 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:28:56.918399    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:29:08.287094 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.287057    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:08.359264 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.359216    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-6nw59"]
Sep 11 16:29:08.359536 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.359485    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f55cd5c49-6nw59" podUID="78cb444b-65ba-4b70-ba7b-8d0019e2e043" containerName="manager" containerID="cri-o://b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37" gracePeriod=30
Sep 11 16:29:08.598201 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.598173    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:29:08.760921 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.760889    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-webhook-cert\") pod \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") "
Sep 11 16:29:08.761090 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.760994    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-metrics-certs\") pod \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") "
Sep 11 16:29:08.761090 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.761024    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfhk2\" (UniqueName: \"kubernetes.io/projected/78cb444b-65ba-4b70-ba7b-8d0019e2e043-kube-api-access-wfhk2\") pod \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\" (UID: \"78cb444b-65ba-4b70-ba7b-8d0019e2e043\") "
Sep 11 16:29:08.763045 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.763012    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "78cb444b-65ba-4b70-ba7b-8d0019e2e043" (UID: "78cb444b-65ba-4b70-ba7b-8d0019e2e043"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:29:08.763166 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.763080    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cb444b-65ba-4b70-ba7b-8d0019e2e043-kube-api-access-wfhk2" (OuterVolumeSpecName: "kube-api-access-wfhk2") pod "78cb444b-65ba-4b70-ba7b-8d0019e2e043" (UID: "78cb444b-65ba-4b70-ba7b-8d0019e2e043"). InnerVolumeSpecName "kube-api-access-wfhk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:29:08.763166 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.763079    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "78cb444b-65ba-4b70-ba7b-8d0019e2e043" (UID: "78cb444b-65ba-4b70-ba7b-8d0019e2e043"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:29:08.862193 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.862143    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:29:08.862193 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.862163    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfhk2\" (UniqueName: \"kubernetes.io/projected/78cb444b-65ba-4b70-ba7b-8d0019e2e043-kube-api-access-wfhk2\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:29:08.862193 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:08.862173    2564 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78cb444b-65ba-4b70-ba7b-8d0019e2e043-webhook-cert\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:29:09.023546 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.023518    2564 generic.go:356] "Generic (PLEG): container finished" podID="78cb444b-65ba-4b70-ba7b-8d0019e2e043" containerID="b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37" exitCode=0
Sep 11 16:29:09.023667 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.023573    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-6nw59"
Sep 11 16:29:09.023667 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.023580    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-6nw59" event={"ID":"78cb444b-65ba-4b70-ba7b-8d0019e2e043","Type":"ContainerDied","Data":"b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37"}
Sep 11 16:29:09.023667 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.023638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-6nw59" event={"ID":"78cb444b-65ba-4b70-ba7b-8d0019e2e043","Type":"ContainerDied","Data":"7145928c6e49ddf7569b8135d5294ba22fce167b602354723448f21995d02e12"}
Sep 11 16:29:09.023667 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.023658    2564 scope.go:117] "RemoveContainer" containerID="b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37"
Sep 11 16:29:09.031591 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.031574    2564 scope.go:117] "RemoveContainer" containerID="b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37"
Sep 11 16:29:09.034152 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:29:09.034126    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37\": container with ID starting with b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37 not found: ID does not exist" containerID="b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37"
Sep 11 16:29:09.034275 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.034160    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37"} err="failed to get container status \"b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37\": rpc error: code = NotFound desc = could not find container \"b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37\": container with ID starting with b9676ebd03ac2bc35ae380543dcb13b159c1e2c23d50c2a9464b7cfe801a9f37 not found: ID does not exist"
Sep 11 16:29:09.047377 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.047349    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-6nw59"]
Sep 11 16:29:09.051554 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.051534    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-6nw59"]
Sep 11 16:29:09.714408 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.714370    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:29:09.818804 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.818776    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f55cd5c49-mtj74"]
Sep 11 16:29:09.819167 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.819149    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78cb444b-65ba-4b70-ba7b-8d0019e2e043" containerName="manager"
Sep 11 16:29:09.819276 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.819169    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78cb444b-65ba-4b70-ba7b-8d0019e2e043" containerName="manager"
Sep 11 16:29:09.819276 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.819262    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78cb444b-65ba-4b70-ba7b-8d0019e2e043" containerName="manager"
Sep 11 16:29:09.823533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.823512    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:09.827702 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.827684    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 16:29:09.828019 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.828005    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 16:29:09.828082 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.828015    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 16:29:09.828082 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.828024    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w4bfh\""
Sep 11 16:29:09.833981 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.833961    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-mtj74"]
Sep 11 16:29:09.972219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.972149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac22be16-1ee0-4cff-8951-fa5e8877c98a-metrics-certs\") pod \"maas-controller-f55cd5c49-mtj74\" (UID: \"ac22be16-1ee0-4cff-8951-fa5e8877c98a\") " pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:09.972219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.972186    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6689n\" (UniqueName: \"kubernetes.io/projected/ac22be16-1ee0-4cff-8951-fa5e8877c98a-kube-api-access-6689n\") pod \"maas-controller-f55cd5c49-mtj74\" (UID: \"ac22be16-1ee0-4cff-8951-fa5e8877c98a\") " pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:09.972422 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:09.972229    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac22be16-1ee0-4cff-8951-fa5e8877c98a-webhook-cert\") pod \"maas-controller-f55cd5c49-mtj74\" (UID: \"ac22be16-1ee0-4cff-8951-fa5e8877c98a\") " pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:10.073225 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:10.073198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac22be16-1ee0-4cff-8951-fa5e8877c98a-webhook-cert\") pod \"maas-controller-f55cd5c49-mtj74\" (UID: \"ac22be16-1ee0-4cff-8951-fa5e8877c98a\") " pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:10.073373 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:10.073289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac22be16-1ee0-4cff-8951-fa5e8877c98a-metrics-certs\") pod \"maas-controller-f55cd5c49-mtj74\" (UID: \"ac22be16-1ee0-4cff-8951-fa5e8877c98a\") " pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:10.073373 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:10.073315    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6689n\" (UniqueName: \"kubernetes.io/projected/ac22be16-1ee0-4cff-8951-fa5e8877c98a-kube-api-access-6689n\") pod \"maas-controller-f55cd5c49-mtj74\" (UID: \"ac22be16-1ee0-4cff-8951-fa5e8877c98a\") " pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:10.075755 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:10.075731    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ac22be16-1ee0-4cff-8951-fa5e8877c98a-webhook-cert\") pod \"maas-controller-f55cd5c49-mtj74\" (UID: \"ac22be16-1ee0-4cff-8951-fa5e8877c98a\") " pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:10.075859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:10.075731    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac22be16-1ee0-4cff-8951-fa5e8877c98a-metrics-certs\") pod \"maas-controller-f55cd5c49-mtj74\" (UID: \"ac22be16-1ee0-4cff-8951-fa5e8877c98a\") " pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:10.082116 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:10.082096    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6689n\" (UniqueName: \"kubernetes.io/projected/ac22be16-1ee0-4cff-8951-fa5e8877c98a-kube-api-access-6689n\") pod \"maas-controller-f55cd5c49-mtj74\" (UID: \"ac22be16-1ee0-4cff-8951-fa5e8877c98a\") " pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:10.133539 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:10.133519    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:10.405712 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:10.405675    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f55cd5c49-mtj74"]
Sep 11 16:29:10.408019 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:29:10.407988    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac22be16_1ee0_4cff_8951_fa5e8877c98a.slice/crio-d1d5b6df324a5a70d8d82c1a25b9556d59e87a0028a172a68e85ca2015bd73a2 WatchSource:0}: Error finding container d1d5b6df324a5a70d8d82c1a25b9556d59e87a0028a172a68e85ca2015bd73a2: Status 404 returned error can't find the container with id d1d5b6df324a5a70d8d82c1a25b9556d59e87a0028a172a68e85ca2015bd73a2
Sep 11 16:29:10.809823 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:10.809749    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78cb444b-65ba-4b70-ba7b-8d0019e2e043" path="/var/lib/kubelet/pods/78cb444b-65ba-4b70-ba7b-8d0019e2e043/volumes"
Sep 11 16:29:11.034665 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:11.034635    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-mtj74" event={"ID":"ac22be16-1ee0-4cff-8951-fa5e8877c98a","Type":"ContainerStarted","Data":"e4f23696d358d39f0dd365d1c7eeb2b9de3df0a69be66f38cba1a3483f5de4ce"}
Sep 11 16:29:11.034805 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:11.034694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f55cd5c49-mtj74" event={"ID":"ac22be16-1ee0-4cff-8951-fa5e8877c98a","Type":"ContainerStarted","Data":"d1d5b6df324a5a70d8d82c1a25b9556d59e87a0028a172a68e85ca2015bd73a2"}
Sep 11 16:29:11.034805 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:11.034775    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:11.056749 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:11.056707    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f55cd5c49-mtj74" podStartSLOduration=1.610266145 podStartE2EDuration="2.056695066s" totalImagesPullingTime="446.428921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:29:11.054288437 +0000 UTC m=+2044.772263582" watchObservedRunningTime="2026-09-11 16:29:11.056695066 +0000 UTC m=+2044.774670186"
Sep 11 16:29:22.041513 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:22.041467    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f55cd5c49-mtj74"
Sep 11 16:29:29.089595 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:29:29.089503    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:30:13.302284 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:13.302235    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"]
Sep 11 16:30:13.302754 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:13.302492    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw" podUID="8d5d595f-0fda-4617-bc08-3cf3c4676fac" containerName="manager" containerID="cri-o://8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5" gracePeriod=10
Sep 11 16:30:14.942379 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:14.942354    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:30:15.032499 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.032432    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2rp9\" (UniqueName: \"kubernetes.io/projected/8d5d595f-0fda-4617-bc08-3cf3c4676fac-kube-api-access-s2rp9\") pod \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\" (UID: \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\") "
Sep 11 16:30:15.032499 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.032476    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d5d595f-0fda-4617-bc08-3cf3c4676fac-extensions-socket-volume\") pod \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\" (UID: \"8d5d595f-0fda-4617-bc08-3cf3c4676fac\") "
Sep 11 16:30:15.032880 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.032858    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d5d595f-0fda-4617-bc08-3cf3c4676fac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8d5d595f-0fda-4617-bc08-3cf3c4676fac" (UID: "8d5d595f-0fda-4617-bc08-3cf3c4676fac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 16:30:15.034364 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.034345    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5d595f-0fda-4617-bc08-3cf3c4676fac-kube-api-access-s2rp9" (OuterVolumeSpecName: "kube-api-access-s2rp9") pod "8d5d595f-0fda-4617-bc08-3cf3c4676fac" (UID: "8d5d595f-0fda-4617-bc08-3cf3c4676fac"). InnerVolumeSpecName "kube-api-access-s2rp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:30:15.133612 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.133582    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2rp9\" (UniqueName: \"kubernetes.io/projected/8d5d595f-0fda-4617-bc08-3cf3c4676fac-kube-api-access-s2rp9\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:30:15.133612 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.133607    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d5d595f-0fda-4617-bc08-3cf3c4676fac-extensions-socket-volume\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:30:15.314220 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.314194    2564 generic.go:356] "Generic (PLEG): container finished" podID="8d5d595f-0fda-4617-bc08-3cf3c4676fac" containerID="8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5" exitCode=0
Sep 11 16:30:15.314339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.314226    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw" event={"ID":"8d5d595f-0fda-4617-bc08-3cf3c4676fac","Type":"ContainerDied","Data":"8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5"}
Sep 11 16:30:15.314339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.314249    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"
Sep 11 16:30:15.314339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.314291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw" event={"ID":"8d5d595f-0fda-4617-bc08-3cf3c4676fac","Type":"ContainerDied","Data":"034dcf959daf999a3cae328752100df153713c1ed2b7dbd1c261f70d0ffa022e"}
Sep 11 16:30:15.314339 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.314314    2564 scope.go:117] "RemoveContainer" containerID="8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5"
Sep 11 16:30:15.323309 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.323293    2564 scope.go:117] "RemoveContainer" containerID="8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5"
Sep 11 16:30:15.323589 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:30:15.323570    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5\": container with ID starting with 8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5 not found: ID does not exist" containerID="8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5"
Sep 11 16:30:15.323641 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.323597    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5"} err="failed to get container status \"8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5\": rpc error: code = NotFound desc = could not find container \"8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5\": container with ID starting with 8e1c0432ea300266774f67c45b8d9e4097ed41a394ed8ee4e13b05bf253cb1d5 not found: ID does not exist"
Sep 11 16:30:15.338021 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.337998    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"]
Sep 11 16:30:15.342433 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:15.342413    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-htwmw"]
Sep 11 16:30:16.810826 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:30:16.810790    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5d595f-0fda-4617-bc08-3cf3c4676fac" path="/var/lib/kubelet/pods/8d5d595f-0fda-4617-bc08-3cf3c4676fac/volumes"
Sep 11 16:31:14.009047 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.009008    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"]
Sep 11 16:31:14.009672 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.009518    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5d595f-0fda-4617-bc08-3cf3c4676fac" containerName="manager"
Sep 11 16:31:14.009672 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.009539    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5d595f-0fda-4617-bc08-3cf3c4676fac" containerName="manager"
Sep 11 16:31:14.009672 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.009639    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5d595f-0fda-4617-bc08-3cf3c4676fac" containerName="manager"
Sep 11 16:31:14.012585 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.012560    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:14.017367 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.017344    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xcpqd\""
Sep 11 16:31:14.038173 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.038147    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"]
Sep 11 16:31:14.067651 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.067629    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctnjq\" (UniqueName: \"kubernetes.io/projected/62b610c7-ff51-4559-b91f-f3501312dbac-kube-api-access-ctnjq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ztrgf\" (UID: \"62b610c7-ff51-4559-b91f-f3501312dbac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:14.067751 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.067698    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/62b610c7-ff51-4559-b91f-f3501312dbac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ztrgf\" (UID: \"62b610c7-ff51-4559-b91f-f3501312dbac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:14.168812 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.168782    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/62b610c7-ff51-4559-b91f-f3501312dbac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ztrgf\" (UID: \"62b610c7-ff51-4559-b91f-f3501312dbac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:14.168919 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.168822    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctnjq\" (UniqueName: \"kubernetes.io/projected/62b610c7-ff51-4559-b91f-f3501312dbac-kube-api-access-ctnjq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ztrgf\" (UID: \"62b610c7-ff51-4559-b91f-f3501312dbac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:14.169122 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.169101    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/62b610c7-ff51-4559-b91f-f3501312dbac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ztrgf\" (UID: \"62b610c7-ff51-4559-b91f-f3501312dbac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:14.181500 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.181476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctnjq\" (UniqueName: \"kubernetes.io/projected/62b610c7-ff51-4559-b91f-f3501312dbac-kube-api-access-ctnjq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ztrgf\" (UID: \"62b610c7-ff51-4559-b91f-f3501312dbac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:14.322920 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.322896    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:14.606990 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:31:14.606961    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62b610c7_ff51_4559_b91f_f3501312dbac.slice/crio-0c16f48202eaacfeaf399d77a6b671ef7a6af6ea0212bae07ff6b66224547ec3 WatchSource:0}: Error finding container 0c16f48202eaacfeaf399d77a6b671ef7a6af6ea0212bae07ff6b66224547ec3: Status 404 returned error can't find the container with id 0c16f48202eaacfeaf399d77a6b671ef7a6af6ea0212bae07ff6b66224547ec3
Sep 11 16:31:14.609839 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:14.609733    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"]
Sep 11 16:31:15.598022 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:15.597986    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf" event={"ID":"62b610c7-ff51-4559-b91f-f3501312dbac","Type":"ContainerStarted","Data":"77946b0ef2e829d75c706f12d0a1d66a311609af1dfc684473b1aa8de85a82fd"}
Sep 11 16:31:15.598453 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:15.598047    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf" event={"ID":"62b610c7-ff51-4559-b91f-f3501312dbac","Type":"ContainerStarted","Data":"0c16f48202eaacfeaf399d77a6b671ef7a6af6ea0212bae07ff6b66224547ec3"}
Sep 11 16:31:15.598453 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:15.598077    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:15.643841 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:15.643791    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf" podStartSLOduration=2.643776379 podStartE2EDuration="2.643776379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:31:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:31:15.640943889 +0000 UTC m=+2169.358919056" watchObservedRunningTime="2026-09-11 16:31:15.643776379 +0000 UTC m=+2169.361751498"
Sep 11 16:31:26.606789 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:26.606751    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ztrgf"
Sep 11 16:31:55.931429 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:55.931396    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:31:57.504324 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:31:57.504286    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:05.427147 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:05.427112    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:24.588311 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:24.588275    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:26.400767 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:26.400727    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:32:43.174599 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.174565    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"]
Sep 11 16:32:43.177729 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.177702    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.197899 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.197831    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"]
Sep 11 16:32:43.207533 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.207505    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cthm\" (UniqueName: \"kubernetes.io/projected/ba27df6e-cfbb-4773-aa39-414307e03273-kube-api-access-6cthm\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.207684 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.207553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba27df6e-cfbb-4773-aa39-414307e03273-metrics-certs\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.207684 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.207657    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba27df6e-cfbb-4773-aa39-414307e03273-openshift-service-ca\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.207850 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.207708    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba27df6e-cfbb-4773-aa39-414307e03273-maas-api-tls\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.308908 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.308875    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba27df6e-cfbb-4773-aa39-414307e03273-metrics-certs\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.309100 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.308949    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba27df6e-cfbb-4773-aa39-414307e03273-openshift-service-ca\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.309100 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.308984    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba27df6e-cfbb-4773-aa39-414307e03273-maas-api-tls\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.309100 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.309033    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cthm\" (UniqueName: \"kubernetes.io/projected/ba27df6e-cfbb-4773-aa39-414307e03273-kube-api-access-6cthm\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.310020 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.309969    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba27df6e-cfbb-4773-aa39-414307e03273-openshift-service-ca\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.312281 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.312218    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba27df6e-cfbb-4773-aa39-414307e03273-metrics-certs\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.312400 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.312334    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba27df6e-cfbb-4773-aa39-414307e03273-maas-api-tls\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.318661 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.318639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cthm\" (UniqueName: \"kubernetes.io/projected/ba27df6e-cfbb-4773-aa39-414307e03273-kube-api-access-6cthm\") pod \"maas-api-984bf6b7d-v5v26\" (UID: \"ba27df6e-cfbb-4773-aa39-414307e03273\") " pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.488081 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.488011    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:43.776992 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.776905    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"]
Sep 11 16:32:43.779835 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:32:43.779802    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba27df6e_cfbb_4773_aa39_414307e03273.slice/crio-8c7cce3565b2e67b96b46727353303210ffa0d3b63b1be72058e3e237cfdb1e0 WatchSource:0}: Error finding container 8c7cce3565b2e67b96b46727353303210ffa0d3b63b1be72058e3e237cfdb1e0: Status 404 returned error can't find the container with id 8c7cce3565b2e67b96b46727353303210ffa0d3b63b1be72058e3e237cfdb1e0
Sep 11 16:32:43.993958 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:43.993926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26" event={"ID":"ba27df6e-cfbb-4773-aa39-414307e03273","Type":"ContainerStarted","Data":"8c7cce3565b2e67b96b46727353303210ffa0d3b63b1be72058e3e237cfdb1e0"}
Sep 11 16:32:44.999601 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:44.999557    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26" event={"ID":"ba27df6e-cfbb-4773-aa39-414307e03273","Type":"ContainerStarted","Data":"3a03305da7b2f8071727c54949383f141b433e6010ee3ae264ba108ff5b9be0a"}
Sep 11 16:32:45.000022 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:44.999623    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:45.022311 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:45.022272    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26" podStartSLOduration=1.4959915129999999 podStartE2EDuration="2.02225791s" totalImagesPullingTime="526.266397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 16:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:32:45.020551868 +0000 UTC m=+2258.738527012" watchObservedRunningTime="2026-09-11 16:32:45.02225791 +0000 UTC m=+2258.740233021"
Sep 11 16:32:51.009859 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:51.009822    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-984bf6b7d-v5v26"
Sep 11 16:32:59.169897 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:32:59.169859    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:00.468728 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:00.468699    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-s49th"]
Sep 11 16:33:00.469227 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:00.468925    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th" podUID="efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" containerName="maas-api" containerID="cri-o://ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1" gracePeriod=30
Sep 11 16:33:01.709936 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.709912    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:33:01.741944 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.741907    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-openshift-service-ca\") pod \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") "
Sep 11 16:33:01.742074 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.741957    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-metrics-certs\") pod \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") "
Sep 11 16:33:01.742074 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.742041    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-maas-api-tls\") pod \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") "
Sep 11 16:33:01.742196 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.742105    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2wnm\" (UniqueName: \"kubernetes.io/projected/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-kube-api-access-k2wnm\") pod \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\" (UID: \"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84\") "
Sep 11 16:33:01.742292 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.742268    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" (UID: "efc49dec-e6b6-44bc-9bf4-bb4b19b24b84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 16:33:01.742407 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.742378    2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-openshift-service-ca\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:33:01.744092 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.744069    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-kube-api-access-k2wnm" (OuterVolumeSpecName: "kube-api-access-k2wnm") pod "efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" (UID: "efc49dec-e6b6-44bc-9bf4-bb4b19b24b84"). InnerVolumeSpecName "kube-api-access-k2wnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 16:33:01.744169 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.744092    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" (UID: "efc49dec-e6b6-44bc-9bf4-bb4b19b24b84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:01.744346 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.744318    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" (UID: "efc49dec-e6b6-44bc-9bf4-bb4b19b24b84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 16:33:01.843081 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.843057    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2wnm\" (UniqueName: \"kubernetes.io/projected/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-kube-api-access-k2wnm\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:33:01.843081 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.843079    2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-metrics-certs\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:33:01.843217 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:01.843089    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84-maas-api-tls\") on node \"ip-10-0-141-101.ec2.internal\" DevicePath \"\""
Sep 11 16:33:02.073409 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.073379    2564 generic.go:356] "Generic (PLEG): container finished" podID="efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" containerID="ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1" exitCode=0
Sep 11 16:33:02.073516 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.073417    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th" event={"ID":"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84","Type":"ContainerDied","Data":"ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1"}
Sep 11 16:33:02.073516 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.073429    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th"
Sep 11 16:33:02.073516 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.073457    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-585675b678-s49th" event={"ID":"efc49dec-e6b6-44bc-9bf4-bb4b19b24b84","Type":"ContainerDied","Data":"a7df10d53345c607e293a6c89415172ccd07890ada7d05747710123a775b997b"}
Sep 11 16:33:02.073516 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.073479    2564 scope.go:117] "RemoveContainer" containerID="ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1"
Sep 11 16:33:02.081988 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.081972    2564 scope.go:117] "RemoveContainer" containerID="ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1"
Sep 11 16:33:02.082260 ip-10-0-141-101 kubenswrapper[2564]: E0911 16:33:02.082220    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1\": container with ID starting with ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1 not found: ID does not exist" containerID="ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1"
Sep 11 16:33:02.082350 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.082263    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1"} err="failed to get container status \"ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1\": rpc error: code = NotFound desc = could not find container \"ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1\": container with ID starting with ffa7dae64681249d01802ba599fb8eb3b51179552a6c3b9b95e078f9c1d268d1 not found: ID does not exist"
Sep 11 16:33:02.102475 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.102422    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-s49th"]
Sep 11 16:33:02.109255 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.109224    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-585675b678-s49th"]
Sep 11 16:33:02.810452 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:02.810422    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" path="/var/lib/kubelet/pods/efc49dec-e6b6-44bc-9bf4-bb4b19b24b84/volumes"
Sep 11 16:33:18.495282 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:18.495229    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hwz4v_0bc70645-7134-43fd-9294-da3a3651d464/manager/0.log"
Sep 11 16:33:18.732733 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:18.732683    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f55cd5c49-mtj74_ac22be16-1ee0-4cff-8951-fa5e8877c98a/manager/0.log"
Sep 11 16:33:18.971989 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:18.971951    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wgdnm_d0dad96b-caaf-4841-99e4-82ceecef8dfb/manager/0.log"
Sep 11 16:33:19.342610 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:19.342584    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-lqgls_8014c0da-410d-409e-8d5f-0bbf23f89b24/copy-manifests/0.log"
Sep 11 16:33:19.363359 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:19.363330    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-lqgls_8014c0da-410d-409e-8d5f-0bbf23f89b24/manager/0.log"
Sep 11 16:33:19.834452 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:19.834429    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-984bf6b7d-v5v26_ba27df6e-cfbb-4773-aa39-414307e03273/maas-api/0.log"
Sep 11 16:33:21.055321 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:21.055285    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-hpf42_ad68438c-75e2-4db3-ae49-316911d70d17/manager/0.log"
Sep 11 16:33:21.164858 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:21.164834    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-ktnfx_a5c46727-f6d3-4480-ac58-258c42f80250/manager/0.log"
Sep 11 16:33:21.395150 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:21.395104    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ztrgf_62b610c7-ff51-4559-b91f-f3501312dbac/manager/0.log"
Sep 11 16:33:22.085826 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:22.085798    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7zbkf_dee3238e-1a57-4905-a3f7-fbaf04baa517/discovery/0.log"
Sep 11 16:33:22.330868 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:22.330839    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c65944c75-l7sqc_95b3495f-b1d2-4d9e-ab6e-0db4d623dd97/kube-auth-proxy/0.log"
Sep 11 16:33:22.691219 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:22.691194    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8slbf_66c21252-78b5-43ab-b332-3922c4d50b79/payload-processing/0.log"
Sep 11 16:33:23.260830 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:23.260800    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-7hflj_c5b11a2e-b9f3-4da4-bf9a-abdffffc111c/main/0.log"
Sep 11 16:33:23.479136 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:23.479111    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-jdflt_5780847c-b0e1-4308-b49f-71980aad43e4/main/0.log"
Sep 11 16:33:23.707464 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:23.707432    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-bf24p_8e2426ef-aac7-447e-8d93-a37f46ba585d/main/0.log"
Sep 11 16:33:28.856602 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:28.856559    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:28.951775 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:28.951745    2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log" limit=1024
Sep 11 16:33:31.849152 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:31.849119    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-74rvc_8844b9e8-33aa-4253-8570-bf7b1b1c346e/global-pull-secret-syncer/0.log"
Sep 11 16:33:32.033148 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:32.033124    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2nz4w_8f7e01ee-e3d6-48c6-b567-ad87583a88bc/konnectivity-agent/0.log"
Sep 11 16:33:32.201958 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:32.201887    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-101.ec2.internal_e8494a3e6a9c170896d58705ab106f0d/haproxy/0.log"
Sep 11 16:33:36.304572 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:36.304539    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-hpf42_ad68438c-75e2-4db3-ae49-316911d70d17/manager/0.log"
Sep 11 16:33:36.327372 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:36.327350    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-ktnfx_a5c46727-f6d3-4480-ac58-258c42f80250/manager/0.log"
Sep 11 16:33:36.466404 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:36.466378    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ztrgf_62b610c7-ff51-4559-b91f-f3501312dbac/manager/0.log"
Sep 11 16:33:38.581254 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:38.581216    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57bcd8f66d-qcgg8_e1b7b75c-4b5b-424f-906d-3afba77b7e1a/metrics-server/0.log"
Sep 11 16:33:38.952529 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:38.952435    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nszbh_a5032b63-67a2-4f5f-8b1a-e491ac67a1a5/node-exporter/0.log"
Sep 11 16:33:38.982666 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:38.982636    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nszbh_a5032b63-67a2-4f5f-8b1a-e491ac67a1a5/kube-rbac-proxy/0.log"
Sep 11 16:33:39.017041 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:39.017020    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nszbh_a5032b63-67a2-4f5f-8b1a-e491ac67a1a5/init-textfile/0.log"
Sep 11 16:33:42.032329 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.032290    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"]
Sep 11 16:33:42.032721 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.032630    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" containerName="maas-api"
Sep 11 16:33:42.032721 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.032642    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" containerName="maas-api"
Sep 11 16:33:42.032721 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.032702    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc49dec-e6b6-44bc-9bf4-bb4b19b24b84" containerName="maas-api"
Sep 11 16:33:42.035984 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.035957    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.038895 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.038868    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bg9tr\"/\"openshift-service-ca.crt\""
Sep 11 16:33:42.039958 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.039933    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bg9tr\"/\"kube-root-ca.crt\""
Sep 11 16:33:42.040081 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.039957    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bg9tr\"/\"default-dockercfg-lx7vd\""
Sep 11 16:33:42.045805 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.045781    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"]
Sep 11 16:33:42.131217 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.131190    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-lib-modules\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.131337 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.131226    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgmq5\" (UniqueName: \"kubernetes.io/projected/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-kube-api-access-tgmq5\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.131337 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.131310    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-podres\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.131425 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.131345    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-proc\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.131425 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.131363    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-sys\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.231657 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.231622    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-lib-modules\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.231775 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.231663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgmq5\" (UniqueName: \"kubernetes.io/projected/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-kube-api-access-tgmq5\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.231775 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.231689    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-podres\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.231775 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.231708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-proc\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.231775 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.231733    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-sys\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.231950 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.231799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-proc\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.231950 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.231825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-sys\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.231950 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.231832    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-lib-modules\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.231950 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.231825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-podres\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.240373 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.240352    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgmq5\" (UniqueName: \"kubernetes.io/projected/9d9f6229-2ad4-4773-8cd5-15cd5ef374a2-kube-api-access-tgmq5\") pod \"perf-node-gather-daemonset-h9gzv\" (UID: \"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2\") " pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.346179 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.346156    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:42.620831 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:42.620807    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"]
Sep 11 16:33:42.623332 ip-10-0-141-101 kubenswrapper[2564]: W0911 16:33:42.623296    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9d9f6229_2ad4_4773_8cd5_15cd5ef374a2.slice/crio-3d14e9c31046e35074632f2c32d00918cb9945772a751d02fd6523dede79095e WatchSource:0}: Error finding container 3d14e9c31046e35074632f2c32d00918cb9945772a751d02fd6523dede79095e: Status 404 returned error can't find the container with id 3d14e9c31046e35074632f2c32d00918cb9945772a751d02fd6523dede79095e
Sep 11 16:33:43.264875 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:43.264834    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv" event={"ID":"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2","Type":"ContainerStarted","Data":"9c79fa9485b54f46476c835a196f914d332bfc6ecd41a91726542cb6df15abc4"}
Sep 11 16:33:43.265409 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:43.264905    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv" event={"ID":"9d9f6229-2ad4-4773-8cd5-15cd5ef374a2","Type":"ContainerStarted","Data":"3d14e9c31046e35074632f2c32d00918cb9945772a751d02fd6523dede79095e"}
Sep 11 16:33:43.265409 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:43.264954    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:43.290441 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:43.290387    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv" podStartSLOduration=1.2903723440000001 podStartE2EDuration="1.290372344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 16:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 16:33:43.287653594 +0000 UTC m=+2317.005628732" watchObservedRunningTime="2026-09-11 16:33:43.290372344 +0000 UTC m=+2317.008347463"
Sep 11 16:33:44.537633 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:44.537604    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t48lx_495d4158-a486-437d-9a76-1de6cca00d7e/dns/0.log"
Sep 11 16:33:44.559692 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:44.559668    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-t48lx_495d4158-a486-437d-9a76-1de6cca00d7e/kube-rbac-proxy/0.log"
Sep 11 16:33:44.613292 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:44.613269    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9m89b_b8be5b91-0905-4d4a-83b7-4e80f4b3545e/dns-node-resolver/0.log"
Sep 11 16:33:45.221247 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:45.221222    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwf6l_1bf6c33a-d45b-40b1-a1ba-f42381cd72ad/node-ca/0.log"
Sep 11 16:33:46.797860 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:46.797835    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7zbkf_dee3238e-1a57-4905-a3f7-fbaf04baa517/discovery/0.log"
Sep 11 16:33:46.839592 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:46.839563    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c65944c75-l7sqc_95b3495f-b1d2-4d9e-ab6e-0db4d623dd97/kube-auth-proxy/0.log"
Sep 11 16:33:46.969651 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:46.969625    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8slbf_66c21252-78b5-43ab-b332-3922c4d50b79/payload-processing/0.log"
Sep 11 16:33:47.880381 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:47.880353    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-26dlc_96390e07-591b-4c7a-bbe7-fe4b350f254e/serve-healthcheck-canary/0.log"
Sep 11 16:33:48.529172 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:48.529144    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncw2l_d5dc268f-8a0d-4d2c-add9-532213e53a5b/kube-rbac-proxy/0.log"
Sep 11 16:33:48.550686 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:48.550656    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncw2l_d5dc268f-8a0d-4d2c-add9-532213e53a5b/exporter/0.log"
Sep 11 16:33:48.571574 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:48.571552    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ncw2l_d5dc268f-8a0d-4d2c-add9-532213e53a5b/extractor/0.log"
Sep 11 16:33:49.279645 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:49.279616    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bg9tr/perf-node-gather-daemonset-h9gzv"
Sep 11 16:33:50.968137 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:50.968108    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-hwz4v_0bc70645-7134-43fd-9294-da3a3651d464/manager/0.log"
Sep 11 16:33:51.284908 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:51.284882    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-f55cd5c49-mtj74_ac22be16-1ee0-4cff-8951-fa5e8877c98a/manager/0.log"
Sep 11 16:33:51.429538 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:51.429512    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wgdnm_d0dad96b-caaf-4841-99e4-82ceecef8dfb/manager/0.log"
Sep 11 16:33:51.675313 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:51.675290    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-lqgls_8014c0da-410d-409e-8d5f-0bbf23f89b24/manager/0.log"
Sep 11 16:33:51.695121 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:51.695098    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58647b66f4-lqgls_8014c0da-410d-409e-8d5f-0bbf23f89b24/copy-manifests/0.log"
Sep 11 16:33:53.372733 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:53.372707    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7zbkf_dee3238e-1a57-4905-a3f7-fbaf04baa517/discovery/0.log"
Sep 11 16:33:53.413328 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:53.413301    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c65944c75-l7sqc_95b3495f-b1d2-4d9e-ab6e-0db4d623dd97/kube-auth-proxy/0.log"
Sep 11 16:33:53.526155 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:53.526133    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8slbf_66c21252-78b5-43ab-b332-3922c4d50b79/payload-processing/0.log"
Sep 11 16:33:55.563676 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:55.563644    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-7zbkf_dee3238e-1a57-4905-a3f7-fbaf04baa517/discovery/0.log"
Sep 11 16:33:55.597321 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:55.597295    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c65944c75-l7sqc_95b3495f-b1d2-4d9e-ab6e-0db4d623dd97/kube-auth-proxy/0.log"
Sep 11 16:33:55.706208 ip-10-0-141-101 kubenswrapper[2564]: I0911 16:33:55.706179    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-8slbf_66c21252-78b5-43ab-b332-3922c4d50b79/payload-processing/0.log"