Sep 16 21:20:12.321649 ip-10-0-135-209 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 21:20:12.321660 ip-10-0-135-209 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 21:20:12.321668 ip-10-0-135-209 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 21:20:12.321894 ip-10-0-135-209 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 21:20:21.562411 ip-10-0-135-209 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 16 21:20:21.562426 ip-10-0-135-209 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ec1cd44575094def8a902fad85c69c6d --
Sep 16 21:21:53.219287 ip-10-0-135-209 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 21:21:53.666306 ip-10-0-135-209 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 21:21:53.666306 ip-10-0-135-209 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 21:21:53.666306 ip-10-0-135-209 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 21:21:53.666306 ip-10-0-135-209 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 21:21:53.666306 ip-10-0-135-209 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 21:21:53.669075 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.668969    2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 21:21:53.673846 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673806    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:21:53.673964 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673954    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:21:53.673964 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673963    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:21:53.673964 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673967    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673970    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673973    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673977    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673981    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673984    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673987    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673992    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.673997    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674000    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674003    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674006    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674009    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674011    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674014    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674017    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674020    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674023    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674026    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:21:53.674057 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674029    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674031    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674034    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674037    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674040    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674042    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674045    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674048    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674051    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674055    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674059    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674069    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674073    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674076    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674078    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674081    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674084    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674086    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674089    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674091    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:21:53.674520 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674094    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674096    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674099    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674101    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674104    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674106    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674109    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674113    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674116    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674120    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674122    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674125    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674127    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674130    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674134    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674137    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674139    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674142    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674145    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674151    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:21:53.675033 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674155    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674158    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674161    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674164    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674167    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674170    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674173    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674176    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674178    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674181    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674184    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674186    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674189    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674191    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674194    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674197    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674199    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674202    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674206    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674209    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674211    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:21:53.675533 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674214    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674217    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674219    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674222    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674224    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674227    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674230    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674233    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674235    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674238    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674240    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674243    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674246    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674249    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674252    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.674254    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675007    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675017    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675020    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:21:53.676098 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675022    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675025    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675028    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675032    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675034    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675037    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675040    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675043    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675045    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675048    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675051    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675053    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675056    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675058    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675062    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675065    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675067    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675070    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675073    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675076    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:21:53.676639 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675079    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675095    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675098    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675101    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675103    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675106    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675109    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675112    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675115    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675119    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675121    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675124    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675126    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675129    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675132    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675134    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675137    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675140    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675142    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675145    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:21:53.677170 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675148    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675150    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675153    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675155    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675161    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675165    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675168    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675172    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675175    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675178    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675182    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675185    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675187    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675190    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675193    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675196    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675198    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675201    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675203    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675206    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:21:53.677673 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675208    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675212    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675215    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675218    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675221    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675224    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675226    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675230    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675233    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675235    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675238    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675240    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675243    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675246    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675248    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675251    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675253    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675256    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675258    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:21:53.678188 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675261    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675264    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675266    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675270    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675274    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675277    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675280    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675282    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675285    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675287    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675290    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675292    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675295    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675297    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675300    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675303    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675305    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675701    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675707    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675710    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:21:53.678659 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675713    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675716    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675718    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675721    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675724    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675726    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675729    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675732    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675734    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675737    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675739    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675742    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675744    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675748    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675752    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675756    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675759    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675763    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675766    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675769    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:21:53.679199 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675772    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675775    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675777    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675780    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675783    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675785    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675788    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675791    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675793    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675796    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675798    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675801    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675803    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675806    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675809    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675811    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675815    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675818    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675820    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:21:53.679698 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675823    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675825    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675828    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675831    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675834    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675836    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675839    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675842    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675845    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675847    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675850    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675853    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675856    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675858    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675861    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675863    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675866    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675868    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675871    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675873    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:21:53.680186 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675876    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675878    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675881    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675883    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675886    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675889    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675891    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675894    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675896    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675912    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675915    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675917    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675920    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675922    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675925    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675927    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675930    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675933    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675935    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675939    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:21:53.680676 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675942    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675945    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675948    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675950    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675953    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675956    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675959    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675962    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675966    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675969    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675971    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675974    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675976    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675979    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675982    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675984    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.675988    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676063    2567 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676101    2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676107    2567 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 21:21:53.681223 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676112    2567 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676116    2567 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676120    2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676124    2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676128    2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676132    2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676135    2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676139    2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676142    2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676145    2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676149    2567 flags.go:64] FLAG: --cgroup-root=""
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676154    2567 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676157    2567 flags.go:64] FLAG: --client-ca-file=""
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676161    2567 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676165    2567 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676169    2567 flags.go:64] FLAG: --cluster-domain=""
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676173    2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676176    2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676179    2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676183    2567 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676187    2567 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676190    2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676193    2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 21:21:53.681714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676196    2567 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676199    2567 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676202    2567 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676206    2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676209    2567 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676212    2567 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676216    2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676220    2567 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676223    2567 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676225    2567 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676228    2567 flags.go:64] FLAG: --enable-server="true"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676231    2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676236    2567 flags.go:64] FLAG: --event-burst="100"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676239    2567 flags.go:64] FLAG: --event-qps="50"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676242    2567 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676245    2567 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676247    2567 flags.go:64] FLAG: --eviction-hard=""
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676251    2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676254    2567 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676257    2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676262    2567 flags.go:64] FLAG: --eviction-soft=""
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676265    2567 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676268    2567 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676271    2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676275    2567 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 21:21:53.682311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676282    2567 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676285    2567 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676289    2567 flags.go:64] FLAG: --feature-gates=""
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676293    2567 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676296    2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676299    2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676302    2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676305    2567 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676308    2567 flags.go:64] FLAG: --help="false"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676311    2567 flags.go:64] FLAG: --hostname-override="ip-10-0-135-209.ec2.internal"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676315    2567 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676317    2567 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676320    2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676324    2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676327    2567 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676330    2567 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676333    2567 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676336    2567 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676339    2567 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676341    2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676345    2567 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676348    2567 flags.go:64] FLAG: --kube-reserved=""
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676351    2567 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676353    2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 21:21:53.682955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676357    2567 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676360    2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676362    2567 flags.go:64] FLAG: --lock-file=""
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676367    2567 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676370    2567 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676373    2567 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676378    2567 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676382    2567 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676386    2567 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676389    2567 flags.go:64] FLAG: --logging-format="text"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676392    2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676395    2567 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676398    2567 flags.go:64] FLAG: --manifest-url=""
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676401    2567 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676405    2567 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676409    2567 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676412    2567 flags.go:64] FLAG: --max-pods="110"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676415    2567 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676418    2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676421    2567 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676424    2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676427    2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676430    2567 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676433    2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676440    2567 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 21:21:53.683560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676445    2567 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676448    2567 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676451    2567 flags.go:64] FLAG: --pod-cidr=""
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676454    2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676459    2567 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676462    2567 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676465    2567 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676468    2567 flags.go:64] FLAG: --port="10250"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676471    2567 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676474    2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ca190d3aed3c78e3"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676477    2567 flags.go:64] FLAG: --qos-reserved=""
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676482    2567 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676485    2567 flags.go:64] FLAG: --register-node="true"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676488    2567 flags.go:64] FLAG: --register-with-taints=""
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676492    2567 flags.go:64] FLAG: --registry-burst="10"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676496    2567 flags.go:64] FLAG: --registry-qps="5"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676499    2567 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676502    2567 flags.go:64] FLAG: --reserved-memory=""
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676506    2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676509    2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676512    2567 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676515    2567 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676518    2567 flags.go:64] FLAG: --runonce="false"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676521    2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676524    2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 21:21:53.684201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676527    2567 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676530    2567 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676533    2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676536    2567 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676539    2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676542    2567 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676545    2567 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676548    2567 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676550    2567 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676553    2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676556    2567 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676559    2567 flags.go:64] FLAG: --system-cgroups=""
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676562    2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676568    2567 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676571    2567 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676574    2567 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676578    2567 flags.go:64] FLAG: --tls-min-version=""
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676581    2567 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676586    2567 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676589    2567 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676592    2567 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676595    2567 flags.go:64] FLAG: --v="2"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676601    2567 flags.go:64] FLAG: --version="false"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676605    2567 flags.go:64] FLAG: --vmodule=""
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676614    2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 21:21:53.684848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.676617    2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676729    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676733    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676736    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676740    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676744    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676748    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676751    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676754    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676757    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676760    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676763    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676766    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676769    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676772    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676775    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676778    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676781    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676784    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:21:53.685499 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676787    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676790    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676793    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676796    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676798    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676801    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676805    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676807    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676810    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676813    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676817    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676819    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676823    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676825    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676828    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676831    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676833    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676836    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676840    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:21:53.685991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676844    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676846    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676849    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676851    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676854    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676857    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676859    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676862    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676865    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676867    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676870    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676873    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676875    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676878    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676881    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676883    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676886    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676889    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676891    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676896    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:21:53.686475 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676913    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676916    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676918    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676922    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676925    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676927    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676930    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676933    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676936    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676939    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676941    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676944    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676946    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676949    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676952    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676954    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676957    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676959    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676962    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676964    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:21:53.687073 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676967    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676969    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676972    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676975    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676977    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676980    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676982    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676985    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676988    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676990    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676993    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676996    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.676999    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.677002    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.677005    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.677009    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.677012    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.677015    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.677019    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:21:53.687563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.677026    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.677029    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.677032    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.677796    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.685543    2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.685560    2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.685582    2567 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.685587    2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685647    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685652    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685655    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685659    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685662    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685665    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:21:53.688055 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685668    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685671    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685674    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685677    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685681    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685684    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685687    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685690    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685693    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685696    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685699    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685701    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685704    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685707    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685709    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685712    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685715    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685717    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685721    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685724    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:21:53.688421 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685728    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685732    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685735    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685739    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685742    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685745    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685748    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685751    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685753    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685756    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685758    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685761    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685764    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685766    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685769    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685777    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685780    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685783    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685785    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685788    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:21:53.688948 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685791    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685794    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685796    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685799    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685802    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685805    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685807    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685810    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685813    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685815    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685818    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685822    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685824    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685827    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685830    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685833    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685836    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685838    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685841    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685843    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:21:53.689439 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685846    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685849    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685851    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685854    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685856    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685859    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685862    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685864    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685867    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685870    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685872    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685875    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685879    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685882    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685885    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685888    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685890    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685893    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685895    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:21:53.690005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685910    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685913    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685916    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685919    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685922    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685925    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685928    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685933    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685936    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685940    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685943    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685946    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685950    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.685953    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.685958    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 21:21:53.690465 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686114    2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686118    2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686122    2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686124    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686127    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686130    2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686134    2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686137    2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686140    2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686143    2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686146    2567 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686149    2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686152    2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686154    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686157    2567 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686159    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686162    2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686165    2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686167    2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 21:21:53.690844 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686170    2567 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686173    2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686175    2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686179    2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686182    2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686184    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686188    2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686190    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686193    2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686195    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686198    2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686201    2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686203    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686206    2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686208    2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686211    2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686214    2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686216    2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686219    2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686222    2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 21:21:53.691348 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686224    2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686227    2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686230    2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686232    2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686235    2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686237    2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686240    2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686242    2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686245    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686247    2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686250    2567 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686253    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686255    2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686258    2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686260    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686264    2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686268    2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686271    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686274    2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 21:21:53.691833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686277    2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686280    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686283    2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686285    2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686288    2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686290    2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686293    2567 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686295    2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686298    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686301    2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686304    2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686306    2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686309    2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686311    2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686314    2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686317    2567 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686319    2567 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686322    2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686325    2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686327    2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 21:21:53.692321 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686330    2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686332    2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686335    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686337    2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686340    2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686342    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686345    2567 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686347    2567 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686350    2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686353    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686355    2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686364    2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686367    2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686369    2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686372    2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686375    2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686378    2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686381    2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686384    2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686386    2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 21:21:53.692807 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:53.686389    2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 21:21:53.693342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.686394    2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 21:21:53.693342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.686515    2567 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 21:21:53.693342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.688955    2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 21:21:53.693342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.690100    2567 server.go:1019] "Starting client certificate rotation"
Sep 16 21:21:53.693342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.690207    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 21:21:53.693342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.690250    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 21:21:53.693342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.693050    2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 21:21:53.694572 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.694551    2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 21:21:53.707684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.707665    2567 log.go:25] "Validated CRI v1 runtime API"
Sep 16 21:21:53.711633 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.711614    2567 log.go:25] "Validated CRI v1 image API"
Sep 16 21:21:53.712425 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.712410    2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 21:21:53.714868 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.714849    2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d7b0c67f-c9df-441d-be0a-0258c3ef0427:/dev/nvme0n1p4 fa47bfd1-4a67-4b6e-920c-89d6f4124d91:/dev/nvme0n1p3]
Sep 16 21:21:53.714953 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.714869    2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 21:21:53.720090 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.719975    2567 manager.go:216] Machine: {Timestamp:2026-09-16 21:21:53.719477734 +0000 UTC m=+0.367889715 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102409 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec231063e2c1fbcc26947510bf48b391 SystemUUID:ec231063-e2c1-fbcc-2694-7510bf48b391 BootID:ec1cd445-7509-4def-8a90-2fad85c69c6d Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:ed:64:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:ed:64:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:a1:5e:19:fe:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 21:21:53.720090 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.720085    2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 21:21:53.720209 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.720162    2567 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 21:21:53.721646 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.721625    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 21:21:53.722299 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.722272    2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 21:21:53.722437 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.722302    2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-209.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 21:21:53.723807 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.723797    2567 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 21:21:53.723844 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.723810    2567 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 21:21:53.723844 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.723818    2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 21:21:53.724722 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.724712    2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 21:21:53.726489 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.726479    2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 21:21:53.732320 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.732308    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 21:21:53.732440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.732430    2567 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 21:21:53.732475 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.732459    2567 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 21:21:53.732475 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.732469    2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 21:21:53.732530 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.732484    2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 21:21:53.732530 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.732495    2567 kubelet.go:398] "Adding apiserver pod source"
Sep 16 21:21:53.732530 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.732504    2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 21:21:53.732894 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.732883    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 21:21:53.732962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.732915    2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 21:21:53.735561 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.735545    2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 21:21:53.738258 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.738239    2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 21:21:53.738336 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.738284    2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 21:21:53.740136 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740122    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 21:21:53.740197 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740139    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 21:21:53.740197 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740145    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 21:21:53.740197 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740150    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 21:21:53.740197 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740161    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 21:21:53.740197 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740170    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 21:21:53.740197 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740185    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 21:21:53.740197 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740192    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 21:21:53.740197 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740199    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 21:21:53.740404 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740211    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 21:21:53.740404 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740223    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 21:21:53.740404 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.740228    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 21:21:53.741117 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.741106    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 21:21:53.741155 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.741118    2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 21:21:53.744217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.744182    2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-209.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 21:21:53.744282 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.744230    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 21:21:53.744282 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.744230    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 21:21:53.745055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.745042    2567 server.go:1268] "Started kubelet"
Sep 16 21:21:53.745147 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.745118    2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 21:21:53.745208 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.745169    2567 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 21:21:53.745668 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.745147    2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 21:21:53.746073 ip-10-0-135-209 systemd[1]: Started Kubernetes Kubelet.
Sep 16 21:21:53.747224 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.747207    2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 21:21:53.752821 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.752796    2567 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 21:21:53.753042 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.752930    2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 21:21:53.756107 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.754955    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-209.ec2.internal.18d5ea00400b3299  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-209.ec2.internal,UID:ip-10-0-135-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-209.ec2.internal,},FirstTimestamp:2026-09-16 21:21:53.745015449 +0000 UTC m=+0.393427476,LastTimestamp:2026-09-16 21:21:53.745015449 +0000 UTC m=+0.393427476,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-209.ec2.internal,}"
Sep 16 21:21:53.758807 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.758783    2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 21:21:53.758807 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.758799    2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 21:21:53.758971 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.758945    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:53.759620 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.759604    2567 factory.go:55] Registering systemd factory
Sep 16 21:21:53.759620 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.759623    2567 factory.go:223] Registration of the systemd container factory successfully
Sep 16 21:21:53.759877 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.759859    2567 factory.go:153] Registering CRI-O factory
Sep 16 21:21:53.759877 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.759876    2567 factory.go:223] Registration of the crio container factory successfully
Sep 16 21:21:53.760034 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.759944    2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 21:21:53.760034 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.759963    2567 factory.go:103] Registering Raw factory
Sep 16 21:21:53.760034 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.759978    2567 manager.go:1196] Started watching for new ooms in manager
Sep 16 21:21:53.760187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.760051    2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 21:21:53.760187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.760054    2567 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 21:21:53.760187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.760076    2567 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 21:21:53.760323 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.760193    2567 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 21:21:53.760323 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.760201    2567 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 21:21:53.760414 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.760346    2567 manager.go:349] Starting recovery of all containers
Sep 16 21:21:53.771164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.770975    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46f9j"
Sep 16 21:21:53.771164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.771000    2567 manager.go:354] Recovery completed
Sep 16 21:21:53.771314 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.771279    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 21:21:53.775816 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.775801    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:21:53.776299 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.776283    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-46f9j"
Sep 16 21:21:53.777594 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.777575    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:21:53.777667 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.777609    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:21:53.777667 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.777619    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:21:53.778412 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.778397    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:21:53.778498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.778413    2567 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 21:21:53.778498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.778423    2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 21:21:53.778498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.778441    2567 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 21:21:53.779181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.779160    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:21:53.779256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.779192    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:21:53.779256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.779206    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:21:53.780946 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.780919    2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-209.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 21:21:53.781133 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.781056    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-209.ec2.internal.18d5ea0041fc4ff9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-209.ec2.internal,UID:ip-10-0-135-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-209.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-209.ec2.internal,},FirstTimestamp:2026-09-16 21:21:53.777594361 +0000 UTC m=+0.426006340,LastTimestamp:2026-09-16 21:21:53.777594361 +0000 UTC m=+0.426006340,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-209.ec2.internal,}"
Sep 16 21:21:53.789561 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.789476    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-209.ec2.internal.18d5ea0041fc9bea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-209.ec2.internal,UID:ip-10-0-135-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-209.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-209.ec2.internal,},FirstTimestamp:2026-09-16 21:21:53.777613802 +0000 UTC m=+0.426025782,LastTimestamp:2026-09-16 21:21:53.777613802 +0000 UTC m=+0.426025782,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-209.ec2.internal,}"
Sep 16 21:21:53.799161 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.799072    2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-209.ec2.internal.18d5ea0041fcc190  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-209.ec2.internal,UID:ip-10-0-135-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-209.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-209.ec2.internal,},FirstTimestamp:2026-09-16 21:21:53.77762344 +0000 UTC m=+0.426035419,LastTimestamp:2026-09-16 21:21:53.77762344 +0000 UTC m=+0.426035419,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-209.ec2.internal,}"
Sep 16 21:21:53.807087 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.807004    2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-209.ec2.internal.18d5ea0041fc4ff9\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-209.ec2.internal.18d5ea0041fc4ff9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-209.ec2.internal,UID:ip-10-0-135-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-209.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-209.ec2.internal,},FirstTimestamp:2026-09-16 21:21:53.777594361 +0000 UTC m=+0.426006340,LastTimestamp:2026-09-16 21:21:53.77917986 +0000 UTC m=+0.427591842,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-209.ec2.internal,}"
Sep 16 21:21:53.816763 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.816680    2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-209.ec2.internal.18d5ea0041fc9bea\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-209.ec2.internal.18d5ea0041fc9bea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-209.ec2.internal,UID:ip-10-0-135-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-209.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-209.ec2.internal,},FirstTimestamp:2026-09-16 21:21:53.777613802 +0000 UTC m=+0.426025782,LastTimestamp:2026-09-16 21:21:53.779198917 +0000 UTC m=+0.427610900,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-209.ec2.internal,}"
Sep 16 21:21:53.922896 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.825386    2567 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-209.ec2.internal.18d5ea0041fcc190\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-209.ec2.internal.18d5ea0041fcc190  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-209.ec2.internal,UID:ip-10-0-135-209.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-209.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-209.ec2.internal,},FirstTimestamp:2026-09-16 21:21:53.77762344 +0000 UTC m=+0.426035419,LastTimestamp:2026-09-16 21:21:53.7792111 +0000 UTC m=+0.427623079,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-209.ec2.internal,}"
Sep 16 21:21:53.922896 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.859539    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:53.922896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.921867    2567 policy_none.go:49] "None policy: Start"
Sep 16 21:21:53.922896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.921888    2567 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 21:21:53.922896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.921920    2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 21:21:53.924028 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.924012    2567 policy_none.go:47] "Start"
Sep 16 21:21:53.934048 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.934015    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 21:21:53.935242 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.935225    2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 21:21:53.935336 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.935273    2567 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 21:21:53.935336 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.935302    2567 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 21:21:53.935428 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.935403    2567 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 16 21:21:53.944471 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.944450    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:21:53.957459 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.957443    2567 manager.go:337] "Starting Device Plugin manager"
Sep 16 21:21:53.957537 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.957479    2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 21:21:53.957537 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.957492    2567 server.go:87] "Starting device plugin registration server"
Sep 16 21:21:53.957828 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.957814    2567 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 21:21:53.957931 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.957826    2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 21:21:53.957985 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.957942    2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 21:21:53.958037 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.958025    2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 21:21:53.958084 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:53.958039    2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 21:21:53.958536 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.958516    2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 21:21:53.958611 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.958572    2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:53.986120 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:53.986100    2567 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-209.ec2.internal\" not found" node="ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.036389 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.036351    2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal"]
Sep 16 21:21:54.036534 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.036441    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:21:54.038085 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.038069    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:21:54.038175 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.038102    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:21:54.038175 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.038119    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:21:54.040398 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.040384    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:21:54.040544 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.040528    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.040592 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.040558    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:21:54.041243 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.041228    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:21:54.041317 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.041256    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:21:54.041317 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.041271    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:21:54.041984 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.041966    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:21:54.042082 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.041996    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:21:54.042082 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.042010    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:21:54.044181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.044166    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.044267 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.044196    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:21:54.044881 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.044866    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:21:54.044969 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.044886    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:21:54.044969 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.044895    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:21:54.058242 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.058223    2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 21:21:54.058992 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.058977    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 21:21:54.059043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.059003    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 21:21:54.059043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.059013    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeHasSufficientPID"
Sep 16 21:21:54.059043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.059034    2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.064475 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.064461    2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.064525 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.064482    2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-209.ec2.internal\": node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:54.071960 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.071937    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-209.ec2.internal\" not found" node="ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.076625 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.076608    2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-209.ec2.internal\" not found" node="ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.087618 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.087601    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:54.161333 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.161294    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2885e0b22b161a32ae54bba2bc4adfb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal\" (UID: \"2885e0b22b161a32ae54bba2bc4adfb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.161333 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.161334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2885e0b22b161a32ae54bba2bc4adfb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal\" (UID: \"2885e0b22b161a32ae54bba2bc4adfb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.161494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.161352    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9292673e818dc6000ad469c880116e19-config\") pod \"kube-apiserver-proxy-ip-10-0-135-209.ec2.internal\" (UID: \"9292673e818dc6000ad469c880116e19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.188459 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.188409    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:54.262098 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.262062    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2885e0b22b161a32ae54bba2bc4adfb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal\" (UID: \"2885e0b22b161a32ae54bba2bc4adfb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.262098 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.262102    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2885e0b22b161a32ae54bba2bc4adfb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal\" (UID: \"2885e0b22b161a32ae54bba2bc4adfb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.262217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.262168    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9292673e818dc6000ad469c880116e19-config\") pod \"kube-apiserver-proxy-ip-10-0-135-209.ec2.internal\" (UID: \"9292673e818dc6000ad469c880116e19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.262217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.262188    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2885e0b22b161a32ae54bba2bc4adfb1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal\" (UID: \"2885e0b22b161a32ae54bba2bc4adfb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.262217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.262185    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2885e0b22b161a32ae54bba2bc4adfb1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal\" (UID: \"2885e0b22b161a32ae54bba2bc4adfb1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.262301 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.262224    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9292673e818dc6000ad469c880116e19-config\") pod \"kube-apiserver-proxy-ip-10-0-135-209.ec2.internal\" (UID: \"9292673e818dc6000ad469c880116e19\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.289255 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.289221    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:54.375857 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.375810    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.381139 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.381109    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.389838 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.389814    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:54.491583 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.491479    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:54.592538 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.592498    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:54.677018 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.676977    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:21:54.690020 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.689993    2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 21:21:54.690138 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.690120    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 21:21:54.690203 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.690172    2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 21:21:54.693402 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:54.693380    2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-209.ec2.internal\" not found"
Sep 16 21:21:54.786990 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.786952    2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:21:54.860578 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.860532    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.874437 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.874406    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 21:21:54.875653 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.875638    2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal"
Sep 16 21:21:54.884557 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.884537    2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 21:21:54.923017 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.922962    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 21:16:53 +0000 UTC" deadline="2028-05-15 06:58:35.17601531 +0000 UTC"
Sep 16 21:21:54.923017 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:54.923002    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14553h36m40.253016378s"
Sep 16 21:21:55.078012 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.077961    2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:21:55.215355 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:55.215315    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2885e0b22b161a32ae54bba2bc4adfb1.slice/crio-adbbf3b43fdbe1680cd02e0675e60b8da00999e333d76e18dc5fea5cb2d8a5a4 WatchSource:0}: Error finding container adbbf3b43fdbe1680cd02e0675e60b8da00999e333d76e18dc5fea5cb2d8a5a4: Status 404 returned error can't find the container with id adbbf3b43fdbe1680cd02e0675e60b8da00999e333d76e18dc5fea5cb2d8a5a4
Sep 16 21:21:55.215634 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:55.215602    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9292673e818dc6000ad469c880116e19.slice/crio-5a642d560d22fefb93ede245ab2b34f515d47804f3ad9f296721178f78e08adc WatchSource:0}: Error finding container 5a642d560d22fefb93ede245ab2b34f515d47804f3ad9f296721178f78e08adc: Status 404 returned error can't find the container with id 5a642d560d22fefb93ede245ab2b34f515d47804f3ad9f296721178f78e08adc
Sep 16 21:21:55.221224 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.221207    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:21:55.735737 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.735701    2567 apiserver.go:52] "Watching apiserver"
Sep 16 21:21:55.743238 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.743218    2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 21:21:55.743598 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.743577    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-frkfh","openshift-network-operator/iptables-alerter-vkjfs","openshift-ovn-kubernetes/ovnkube-node-579wp","kube-system/global-pull-secret-syncer-dh4v8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr","openshift-cluster-node-tuning-operator/tuned-kh5ps","openshift-image-registry/node-ca-kfcgt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal","openshift-multus/multus-additional-cni-plugins-hgq4m","openshift-multus/multus-d4fm5","kube-system/konnectivity-agent-f247x","kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal","openshift-dns/node-resolver-xd599","openshift-multus/network-metrics-daemon-x98p4"]
Sep 16 21:21:55.747544 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.747512    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:21:55.747628 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.747608    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:21:55.749747 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.749731    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.752186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.752084    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9cc4r\""
Sep 16 21:21:55.752186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.752158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.752186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.752185    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 21:21:55.752434 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.752267    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:21:55.752434 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.752309    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 21:21:55.754509 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.754485    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:55.754631 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.754552    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:21:55.754740 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.754721    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 21:21:55.755808 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.755785    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lhjks\""
Sep 16 21:21:55.755941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.755806    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 21:21:55.755941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.755819    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 21:21:55.755941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.755793    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 21:21:55.755941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.755805    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 21:21:55.755941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.755889    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 21:21:55.756917 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.756894    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.759078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.759058    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 21:21:55.759078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.759074    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 21:21:55.759233 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.759093    2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 21:21:55.759233 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.759147    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sdprx\""
Sep 16 21:21:55.759233 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.759204    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 21:21:55.759405 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.759389    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.761488 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.761469    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:21:55.761699 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.761683    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n5rx9\""
Sep 16 21:21:55.761760 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.761685    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 21:21:55.761933 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.761920    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.763874 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.763856    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 21:21:55.763967 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.763939    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 21:21:55.764305 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.764204    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 21:21:55.764447 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.764429    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9ftm7\""
Sep 16 21:21:55.764869 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.764854    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.767823 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.767446    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.768630 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.767940    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9fflp\""
Sep 16 21:21:55.768630 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.768129    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 21:21:55.768630 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.768297    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 21:21:55.768630 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.768307    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 21:21:55.768630 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.768599    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 21:21:55.769018 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.768967    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 21:21:55.770449 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.769676    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 21:21:55.770449 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.769847    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-22xjm\""
Sep 16 21:21:55.770957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.770939    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9c5t\" (UniqueName: \"kubernetes.io/projected/b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5-kube-api-access-v9c5t\") pod \"node-ca-kfcgt\" (UID: \"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5\") " pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.771081 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.770973    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771081 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771003    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-kubernetes\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.771081 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-run-systemd\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771075    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-run-ovn-kubernetes\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771112    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-cni-netd\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771141    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0909f050-da4b-4976-b90b-18d1f4e3f352-ovnkube-script-lib\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771169    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26787c23-3f06-45d7-bde9-d895e5de9835-tmp\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.771244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771182    2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 21:21:55.771244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771197    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-etc-openvswitch\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771227    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-etc-selinux\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771272    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-sys-fs\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771301    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbb74\" (UniqueName: \"kubernetes.io/projected/5c654f14-60cd-4513-a8cc-9fad85b35314-kube-api-access-kbb74\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771341    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-device-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771373    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/406b94f6-b82c-49f4-b43c-7505af5cee56-host-slash\") pod \"iptables-alerter-vkjfs\" (UID: \"406b94f6-b82c-49f4-b43c-7505af5cee56\") " pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-var-lib-openvswitch\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771428    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5-host\") pod \"node-ca-kfcgt\" (UID: \"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5\") " pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771444    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-systemd-units\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-run-ovn\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771479    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-node-log\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771493    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-cni-bin\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0909f050-da4b-4976-b90b-18d1f4e3f352-ovnkube-config\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771531    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0909f050-da4b-4976-b90b-18d1f4e3f352-env-overrides\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.771551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771547    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0909f050-da4b-4976-b90b-18d1f4e3f352-ovn-node-metrics-cert\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771591    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771620    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771645    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-socket-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-registration-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771708    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5-serviceca\") pod \"node-ca-kfcgt\" (UID: \"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5\") " pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771734    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771754    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6xnf\" (UniqueName: \"kubernetes.io/projected/406b94f6-b82c-49f4-b43c-7505af5cee56-kube-api-access-q6xnf\") pod \"iptables-alerter-vkjfs\" (UID: \"406b94f6-b82c-49f4-b43c-7505af5cee56\") " pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771773    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38c2151a-8d75-4772-a0a2-922fb7ca7e20-kubelet-config\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771793    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38c2151a-8d75-4772-a0a2-922fb7ca7e20-dbus\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771814    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/406b94f6-b82c-49f4-b43c-7505af5cee56-iptables-alerter-script\") pod \"iptables-alerter-vkjfs\" (UID: \"406b94f6-b82c-49f4-b43c-7505af5cee56\") " pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771828    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-slash\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771849    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-run-netns\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771868    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-log-socket\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771890    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-modprobe-d\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771934    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-sysctl-conf\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772023 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.771950    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-kubelet\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772006    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lhv7\" (UniqueName: \"kubernetes.io/projected/0909f050-da4b-4976-b90b-18d1f4e3f352-kube-api-access-9lhv7\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772050    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-sysconfig\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772084    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-sysctl-d\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772111    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-run\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772134    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-sys\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772156    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-lib-modules\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-host\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772229    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-run-openvswitch\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772299    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-systemd\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772324    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-var-lib-kubelet\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772351    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26787c23-3f06-45d7-bde9-d895e5de9835-etc-tuned\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.772685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.772385    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwr9l\" (UniqueName: \"kubernetes.io/projected/26787c23-3f06-45d7-bde9-d895e5de9835-kube-api-access-kwr9l\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.774136 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.774117    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lx29h\""
Sep 16 21:21:55.774211 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.774124    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 21:21:55.774211 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.774194    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 21:21:55.774336 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.774321    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.776282 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.776264    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n2nfl\""
Sep 16 21:21:55.776543 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.776517    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 21:21:55.776676 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.776549    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 21:21:55.776819 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.776806    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:55.776873 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.776858    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:21:55.805232 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.805212    2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-222js"
Sep 16 21:21:55.813131 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.813113    2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-222js"
Sep 16 21:21:55.860782 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.860744    2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 21:21:55.873585 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873558    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5-host\") pod \"node-ca-kfcgt\" (UID: \"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5\") " pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.873750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873594    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-node-log\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.873750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873621    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-host\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.873750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873642    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26787c23-3f06-45d7-bde9-d895e5de9835-tmp\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.873750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873668    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-var-lib-cni-multus\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.873750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873686    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-node-log\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.873750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873687    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5-host\") pod \"node-ca-kfcgt\" (UID: \"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5\") " pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.873750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873709    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-host\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.873750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873724    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5-serviceca\") pod \"node-ca-kfcgt\" (UID: \"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5\") " pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873762    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873847    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xnf\" (UniqueName: \"kubernetes.io/projected/406b94f6-b82c-49f4-b43c-7505af5cee56-kube-api-access-q6xnf\") pod \"iptables-alerter-vkjfs\" (UID: \"406b94f6-b82c-49f4-b43c-7505af5cee56\") " pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.873982    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-log-socket\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-var-lib-kubelet\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874038    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwr9l\" (UniqueName: \"kubernetes.io/projected/26787c23-3f06-45d7-bde9-d895e5de9835-kube-api-access-kwr9l\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874067    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-multus-cni-dir\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtlnb\" (UniqueName: \"kubernetes.io/projected/94aef974-513c-400a-b957-c5b8046ede7d-kube-api-access-mtlnb\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874116    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38c2151a-8d75-4772-a0a2-922fb7ca7e20-kubelet-config\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874137    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-slash\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874144    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-var-lib-kubelet\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874177    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-modprobe-d\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-run\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874226    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-run-openvswitch\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874235    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-slash\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874245    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26787c23-3f06-45d7-bde9-d895e5de9835-etc-tuned\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874267    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-modprobe-d\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874258    2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 21:21:55.874358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874295    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38c2151a-8d75-4772-a0a2-922fb7ca7e20-kubelet-config\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874302    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-run\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874321    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-run-openvswitch\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-system-cni-dir\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874294    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5-serviceca\") pod \"node-ca-kfcgt\" (UID: \"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5\") " pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874359    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-log-socket\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-hostroot\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77e43015-e830-4e21-afed-16ddd27263e9-system-cni-dir\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874521    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-run-netns\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874560    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5tq7\" (UniqueName: \"kubernetes.io/projected/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-kube-api-access-h5tq7\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874602    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-run-ovn-kubernetes\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874647    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-run-ovn-kubernetes\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874643    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-cni-netd\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874688    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-cni-netd\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874688    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-var-lib-cni-bin\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874760    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7386598c-4acd-4326-8f66-f455708aaa53-konnectivity-ca\") pod \"konnectivity-agent-f247x\" (UID: \"7386598c-4acd-4326-8f66-f455708aaa53\") " pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874786    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77e43015-e830-4e21-afed-16ddd27263e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.875335 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874817    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0909f050-da4b-4976-b90b-18d1f4e3f352-env-overrides\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874844    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-sysctl-conf\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874874    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-etc-openvswitch\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874903    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbb74\" (UniqueName: \"kubernetes.io/projected/5c654f14-60cd-4513-a8cc-9fad85b35314-kube-api-access-kbb74\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874951    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-device-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.874979    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-var-lib-openvswitch\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875003    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-cnibin\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875028    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-systemd-units\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875053    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-run-ovn\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875077    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0909f050-da4b-4976-b90b-18d1f4e3f352-ovn-node-metrics-cert\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875105    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-sysconfig\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875131    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-sys\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875156    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-multus-socket-dir-parent\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875184    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfgkk\" (UniqueName: \"kubernetes.io/projected/5fb0449b-4fbe-446e-8aa0-9d2c653d440d-kube-api-access-zfgkk\") pod \"node-resolver-xd599\" (UID: \"5fb0449b-4fbe-446e-8aa0-9d2c653d440d\") " pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875208    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875261    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.876256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-socket-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875324    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-registration-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875348    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-run-netns\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875371    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94aef974-513c-400a-b957-c5b8046ede7d-multus-daemon-config\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875387    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0909f050-da4b-4976-b90b-18d1f4e3f352-env-overrides\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875399    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77e43015-e830-4e21-afed-16ddd27263e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875488    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-sysconfig\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-etc-openvswitch\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875523    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38c2151a-8d75-4772-a0a2-922fb7ca7e20-dbus\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875542    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-var-lib-openvswitch\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.875551    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.875645    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret podName:38c2151a-8d75-4772-a0a2-922fb7ca7e20 nodeName:}" failed. No retries permitted until 2026-09-16 21:21:56.375605927 +0000 UTC m=+3.024017916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret") pod "global-pull-secret-syncer-dh4v8" (UID: "38c2151a-8d75-4772-a0a2-922fb7ca7e20") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875645    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-socket-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875743    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-sys\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875853    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-sysctl-conf\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875554    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/406b94f6-b82c-49f4-b43c-7505af5cee56-iptables-alerter-script\") pod \"iptables-alerter-vkjfs\" (UID: \"406b94f6-b82c-49f4-b43c-7505af5cee56\") " pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875879    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-run-netns\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875899    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-run-ovn\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875899    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-lib-modules\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875921    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-registration-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875939    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-run-multus-certs\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875955    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-device-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.875971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-kubelet\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876004    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38c2151a-8d75-4772-a0a2-922fb7ca7e20-dbus\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876018    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lhv7\" (UniqueName: \"kubernetes.io/projected/0909f050-da4b-4976-b90b-18d1f4e3f352-kube-api-access-9lhv7\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876059    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-kubelet-dir\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876071    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-lib-modules\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876106    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-sysctl-d\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876136    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/406b94f6-b82c-49f4-b43c-7505af5cee56-iptables-alerter-script\") pod \"iptables-alerter-vkjfs\" (UID: \"406b94f6-b82c-49f4-b43c-7505af5cee56\") " pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876137    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-run-k8s-cni-cncf-io\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-kubelet\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876208    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-sysctl-d\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876227    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-systemd-units\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876249    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-etc-kubernetes\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.877942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876290    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77e43015-e830-4e21-afed-16ddd27263e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876339    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb9nt\" (UniqueName: \"kubernetes.io/projected/77e43015-e830-4e21-afed-16ddd27263e9-kube-api-access-nb9nt\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876379    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-systemd\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876406    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94aef974-513c-400a-b957-c5b8046ede7d-cni-binary-copy\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876430    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7386598c-4acd-4326-8f66-f455708aaa53-agent-certs\") pod \"konnectivity-agent-f247x\" (UID: \"7386598c-4acd-4326-8f66-f455708aaa53\") " pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876456    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5fb0449b-4fbe-446e-8aa0-9d2c653d440d-hosts-file\") pod \"node-resolver-xd599\" (UID: \"5fb0449b-4fbe-446e-8aa0-9d2c653d440d\") " pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876491    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9c5t\" (UniqueName: \"kubernetes.io/projected/b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5-kube-api-access-v9c5t\") pod \"node-ca-kfcgt\" (UID: \"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5\") " pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876516    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876545    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-kubernetes\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876554    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-systemd\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876570    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-multus-conf-dir\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876595    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77e43015-e830-4e21-afed-16ddd27263e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876609    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876623    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-run-systemd\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876648    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0909f050-da4b-4976-b90b-18d1f4e3f352-ovnkube-script-lib\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876672    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb0449b-4fbe-446e-8aa0-9d2c653d440d-tmp-dir\") pod \"node-resolver-xd599\" (UID: \"5fb0449b-4fbe-446e-8aa0-9d2c653d440d\") " pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876697    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77e43015-e830-4e21-afed-16ddd27263e9-cnibin\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.878732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876703    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26787c23-3f06-45d7-bde9-d895e5de9835-etc-kubernetes\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876722    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-cni-bin\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876739    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-run-systemd\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876750    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0909f050-da4b-4976-b90b-18d1f4e3f352-ovnkube-config\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876785    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-etc-selinux\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876813    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-sys-fs\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876841    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/406b94f6-b82c-49f4-b43c-7505af5cee56-host-slash\") pod \"iptables-alerter-vkjfs\" (UID: \"406b94f6-b82c-49f4-b43c-7505af5cee56\") " pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.876873    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-var-lib-kubelet\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.877004    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-etc-selinux\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.877057    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5c654f14-60cd-4513-a8cc-9fad85b35314-sys-fs\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.877111    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/406b94f6-b82c-49f4-b43c-7505af5cee56-host-slash\") pod \"iptables-alerter-vkjfs\" (UID: \"406b94f6-b82c-49f4-b43c-7505af5cee56\") " pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.877155    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0909f050-da4b-4976-b90b-18d1f4e3f352-host-cni-bin\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.877323    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0909f050-da4b-4976-b90b-18d1f4e3f352-ovnkube-script-lib\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.877323    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0909f050-da4b-4976-b90b-18d1f4e3f352-ovnkube-config\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.878685    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26787c23-3f06-45d7-bde9-d895e5de9835-tmp\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.878713    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26787c23-3f06-45d7-bde9-d895e5de9835-etc-tuned\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.879474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.878916    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0909f050-da4b-4976-b90b-18d1f4e3f352-ovn-node-metrics-cert\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.889707 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.889679    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6xnf\" (UniqueName: \"kubernetes.io/projected/406b94f6-b82c-49f4-b43c-7505af5cee56-kube-api-access-q6xnf\") pod \"iptables-alerter-vkjfs\" (UID: \"406b94f6-b82c-49f4-b43c-7505af5cee56\") " pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:55.893786 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.893760    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:21:55.893895 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.893794    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:21:55.893895 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.893809    2567 projected.go:196] Error preparing data for projected volume kube-api-access-zhswz for pod openshift-network-diagnostics/network-check-target-frkfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:21:55.893895 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.893880    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz podName:d4a729ab-f653-4c7c-a0bb-23569feae959 nodeName:}" failed. No retries permitted until 2026-09-16 21:21:56.393858905 +0000 UTC m=+3.042270886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zhswz" (UniqueName: "kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz") pod "network-check-target-frkfh" (UID: "d4a729ab-f653-4c7c-a0bb-23569feae959") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:21:55.896194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.896141    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwr9l\" (UniqueName: \"kubernetes.io/projected/26787c23-3f06-45d7-bde9-d895e5de9835-kube-api-access-kwr9l\") pod \"tuned-kh5ps\" (UID: \"26787c23-3f06-45d7-bde9-d895e5de9835\") " pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:55.896333 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.896314    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9c5t\" (UniqueName: \"kubernetes.io/projected/b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5-kube-api-access-v9c5t\") pod \"node-ca-kfcgt\" (UID: \"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5\") " pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:55.898029 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.897962    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lhv7\" (UniqueName: \"kubernetes.io/projected/0909f050-da4b-4976-b90b-18d1f4e3f352-kube-api-access-9lhv7\") pod \"ovnkube-node-579wp\" (UID: \"0909f050-da4b-4976-b90b-18d1f4e3f352\") " pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:55.898434 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.898416    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbb74\" (UniqueName: \"kubernetes.io/projected/5c654f14-60cd-4513-a8cc-9fad85b35314-kube-api-access-kbb74\") pod \"aws-ebs-csi-driver-node-22tnr\" (UID: \"5c654f14-60cd-4513-a8cc-9fad85b35314\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:55.942400 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.942341    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal" event={"ID":"2885e0b22b161a32ae54bba2bc4adfb1","Type":"ContainerStarted","Data":"adbbf3b43fdbe1680cd02e0675e60b8da00999e333d76e18dc5fea5cb2d8a5a4"}
Sep 16 21:21:55.943736 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.943709    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal" event={"ID":"9292673e818dc6000ad469c880116e19","Type":"ContainerStarted","Data":"5a642d560d22fefb93ede245ab2b34f515d47804f3ad9f296721178f78e08adc"}
Sep 16 21:21:55.977378 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977342    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-var-lib-cni-multus\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.977566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-multus-cni-dir\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.977566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977424    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtlnb\" (UniqueName: \"kubernetes.io/projected/94aef974-513c-400a-b957-c5b8046ede7d-kube-api-access-mtlnb\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.977566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-system-cni-dir\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.977566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977456    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-var-lib-cni-multus\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.977566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977472    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-hostroot\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.977566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977497    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77e43015-e830-4e21-afed-16ddd27263e9-system-cni-dir\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.977566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977519    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-run-netns\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.977566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977542    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5tq7\" (UniqueName: \"kubernetes.io/projected/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-kube-api-access-h5tq7\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:55.977566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977565    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-var-lib-cni-bin\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977586    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7386598c-4acd-4326-8f66-f455708aaa53-konnectivity-ca\") pod \"konnectivity-agent-f247x\" (UID: \"7386598c-4acd-4326-8f66-f455708aaa53\") " pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977609    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77e43015-e830-4e21-afed-16ddd27263e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977638    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-cnibin\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977727    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-hostroot\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977750    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-system-cni-dir\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977776    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77e43015-e830-4e21-afed-16ddd27263e9-system-cni-dir\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977837    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77e43015-e830-4e21-afed-16ddd27263e9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977859    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-multus-socket-dir-parent\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfgkk\" (UniqueName: \"kubernetes.io/projected/5fb0449b-4fbe-446e-8aa0-9d2c653d440d-kube-api-access-zfgkk\") pod \"node-resolver-xd599\" (UID: \"5fb0449b-4fbe-446e-8aa0-9d2c653d440d\") " pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977949    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-multus-socket-dir-parent\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977953    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.977961    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-cnibin\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978005    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-run-netns\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978035 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.978035    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978064    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94aef974-513c-400a-b957-c5b8046ede7d-multus-daemon-config\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:55.978099    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs podName:ffed9181-7dab-40e2-aa2a-94f3ef7dae5d nodeName:}" failed. No retries permitted until 2026-09-16 21:21:56.478077792 +0000 UTC m=+3.126489761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs") pod "network-metrics-daemon-x98p4" (UID: "ffed9181-7dab-40e2-aa2a-94f3ef7dae5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978128    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77e43015-e830-4e21-afed-16ddd27263e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978140    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-multus-cni-dir\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978175    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-var-lib-cni-bin\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978210    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-run-multus-certs\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978241    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-run-k8s-cni-cncf-io\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978264    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-etc-kubernetes\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978286    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77e43015-e830-4e21-afed-16ddd27263e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978313    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb9nt\" (UniqueName: \"kubernetes.io/projected/77e43015-e830-4e21-afed-16ddd27263e9-kube-api-access-nb9nt\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978340    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94aef974-513c-400a-b957-c5b8046ede7d-cni-binary-copy\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978362    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7386598c-4acd-4326-8f66-f455708aaa53-agent-certs\") pod \"konnectivity-agent-f247x\" (UID: \"7386598c-4acd-4326-8f66-f455708aaa53\") " pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978368    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-etc-kubernetes\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978385    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5fb0449b-4fbe-446e-8aa0-9d2c653d440d-hosts-file\") pod \"node-resolver-xd599\" (UID: \"5fb0449b-4fbe-446e-8aa0-9d2c653d440d\") " pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978412    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-multus-conf-dir\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978419    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-run-multus-certs\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.978684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978435    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77e43015-e830-4e21-afed-16ddd27263e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978462    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb0449b-4fbe-446e-8aa0-9d2c653d440d-tmp-dir\") pod \"node-resolver-xd599\" (UID: \"5fb0449b-4fbe-446e-8aa0-9d2c653d440d\") " pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77e43015-e830-4e21-afed-16ddd27263e9-cnibin\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978514    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-var-lib-kubelet\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978592    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-var-lib-kubelet\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978612    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7386598c-4acd-4326-8f66-f455708aaa53-konnectivity-ca\") pod \"konnectivity-agent-f247x\" (UID: \"7386598c-4acd-4326-8f66-f455708aaa53\") " pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978646    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5fb0449b-4fbe-446e-8aa0-9d2c653d440d-hosts-file\") pod \"node-resolver-xd599\" (UID: \"5fb0449b-4fbe-446e-8aa0-9d2c653d440d\") " pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978460    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-host-run-k8s-cni-cncf-io\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978674    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77e43015-e830-4e21-afed-16ddd27263e9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978695    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77e43015-e830-4e21-afed-16ddd27263e9-cnibin\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/94aef974-513c-400a-b957-c5b8046ede7d-multus-conf-dir\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.978936    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fb0449b-4fbe-446e-8aa0-9d2c653d440d-tmp-dir\") pod \"node-resolver-xd599\" (UID: \"5fb0449b-4fbe-446e-8aa0-9d2c653d440d\") " pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.979020    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/94aef974-513c-400a-b957-c5b8046ede7d-multus-daemon-config\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.979308    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77e43015-e830-4e21-afed-16ddd27263e9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.979426    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/94aef974-513c-400a-b957-c5b8046ede7d-cni-binary-copy\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:55.979501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.979445    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77e43015-e830-4e21-afed-16ddd27263e9-cni-binary-copy\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.981256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.981235    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7386598c-4acd-4326-8f66-f455708aaa53-agent-certs\") pod \"konnectivity-agent-f247x\" (UID: \"7386598c-4acd-4326-8f66-f455708aaa53\") " pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:21:55.987126 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.987064    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb9nt\" (UniqueName: \"kubernetes.io/projected/77e43015-e830-4e21-afed-16ddd27263e9-kube-api-access-nb9nt\") pod \"multus-additional-cni-plugins-hgq4m\" (UID: \"77e43015-e830-4e21-afed-16ddd27263e9\") " pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:55.987408 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.987387    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5tq7\" (UniqueName: \"kubernetes.io/projected/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-kube-api-access-h5tq7\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:55.987496 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.987413    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfgkk\" (UniqueName: \"kubernetes.io/projected/5fb0449b-4fbe-446e-8aa0-9d2c653d440d-kube-api-access-zfgkk\") pod \"node-resolver-xd599\" (UID: \"5fb0449b-4fbe-446e-8aa0-9d2c653d440d\") " pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:55.987690 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:55.987666    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtlnb\" (UniqueName: \"kubernetes.io/projected/94aef974-513c-400a-b957-c5b8046ede7d-kube-api-access-mtlnb\") pod \"multus-d4fm5\" (UID: \"94aef974-513c-400a-b957-c5b8046ede7d\") " pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:56.060639 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.060592    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vkjfs"
Sep 16 21:21:56.067452 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.067419    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:21:56.076624 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.076597    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr"
Sep 16 21:21:56.082279 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:56.082133    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0909f050_da4b_4976_b90b_18d1f4e3f352.slice/crio-d50e1bcdf6ae6e012db44cc83ed91bd9c18bdddb77f31d15538a38020468ed82 WatchSource:0}: Error finding container d50e1bcdf6ae6e012db44cc83ed91bd9c18bdddb77f31d15538a38020468ed82: Status 404 returned error can't find the container with id d50e1bcdf6ae6e012db44cc83ed91bd9c18bdddb77f31d15538a38020468ed82
Sep 16 21:21:56.082708 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.082687    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kh5ps"
Sep 16 21:21:56.088358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.088339    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kfcgt"
Sep 16 21:21:56.092784 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:56.092753    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c654f14_60cd_4513_a8cc_9fad85b35314.slice/crio-5047606befc104c3c50943196ba933b18a50f3ef5a6f3138b15ff5f1e98a0021 WatchSource:0}: Error finding container 5047606befc104c3c50943196ba933b18a50f3ef5a6f3138b15ff5f1e98a0021: Status 404 returned error can't find the container with id 5047606befc104c3c50943196ba933b18a50f3ef5a6f3138b15ff5f1e98a0021
Sep 16 21:21:56.094096 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.094077    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hgq4m"
Sep 16 21:21:56.097005 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:56.096973    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26787c23_3f06_45d7_bde9_d895e5de9835.slice/crio-2d5ce2a71bdd64fa5d114816f28f61f4d785e58e04b0cfa58da5935c5d3d75ab WatchSource:0}: Error finding container 2d5ce2a71bdd64fa5d114816f28f61f4d785e58e04b0cfa58da5935c5d3d75ab: Status 404 returned error can't find the container with id 2d5ce2a71bdd64fa5d114816f28f61f4d785e58e04b0cfa58da5935c5d3d75ab
Sep 16 21:21:56.099341 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.099323    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d4fm5"
Sep 16 21:21:56.100730 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:56.100706    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45b7d4a_6f16_42a0_b70d_7e63ca3efdd5.slice/crio-6bdc575da2a058459bf8e3ace61c2078255424618a9e6327d3c9ebbf396255d1 WatchSource:0}: Error finding container 6bdc575da2a058459bf8e3ace61c2078255424618a9e6327d3c9ebbf396255d1: Status 404 returned error can't find the container with id 6bdc575da2a058459bf8e3ace61c2078255424618a9e6327d3c9ebbf396255d1
Sep 16 21:21:56.105172 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.105154    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:21:56.109799 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:56.109770    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e43015_e830_4e21_afed_16ddd27263e9.slice/crio-fe1da98b5150f038fb8dfc00d7256efebce9d8a7c8ae9201c45a21fe2ddf0577 WatchSource:0}: Error finding container fe1da98b5150f038fb8dfc00d7256efebce9d8a7c8ae9201c45a21fe2ddf0577: Status 404 returned error can't find the container with id fe1da98b5150f038fb8dfc00d7256efebce9d8a7c8ae9201c45a21fe2ddf0577
Sep 16 21:21:56.110787 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.110770    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xd599"
Sep 16 21:21:56.121201 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:56.120963    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7386598c_4acd_4326_8f66_f455708aaa53.slice/crio-fae512b2ae0dd3c0371e67793a3c5d4b6b654104205c2c1ed66bc88f61551cc9 WatchSource:0}: Error finding container fae512b2ae0dd3c0371e67793a3c5d4b6b654104205c2c1ed66bc88f61551cc9: Status 404 returned error can't find the container with id fae512b2ae0dd3c0371e67793a3c5d4b6b654104205c2c1ed66bc88f61551cc9
Sep 16 21:21:56.123751 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:56.123533    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94aef974_513c_400a_b957_c5b8046ede7d.slice/crio-0600b7872e31e246e00d209db1c3f47483573825439bb5f3957e77e65be25697 WatchSource:0}: Error finding container 0600b7872e31e246e00d209db1c3f47483573825439bb5f3957e77e65be25697: Status 404 returned error can't find the container with id 0600b7872e31e246e00d209db1c3f47483573825439bb5f3957e77e65be25697
Sep 16 21:21:56.126669 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:21:56.126648    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb0449b_4fbe_446e_8aa0_9d2c653d440d.slice/crio-94949115258c177c9ed67bde46b888eb5953e7d07de5090ca1430307f8c61743 WatchSource:0}: Error finding container 94949115258c177c9ed67bde46b888eb5953e7d07de5090ca1430307f8c61743: Status 404 returned error can't find the container with id 94949115258c177c9ed67bde46b888eb5953e7d07de5090ca1430307f8c61743
Sep 16 21:21:56.246591 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.246485    2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:21:56.381559 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.381522    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:56.381738 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.381668    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:21:56.381738 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.381737    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret podName:38c2151a-8d75-4772-a0a2-922fb7ca7e20 nodeName:}" failed. No retries permitted until 2026-09-16 21:21:57.381719405 +0000 UTC m=+4.030131384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret") pod "global-pull-secret-syncer-dh4v8" (UID: "38c2151a-8d75-4772-a0a2-922fb7ca7e20") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:21:56.482362 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.482324    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:21:56.482528 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.482380    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:56.482528 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.482490    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:21:56.482646 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.482532    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:21:56.482646 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.482554    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs podName:ffed9181-7dab-40e2-aa2a-94f3ef7dae5d nodeName:}" failed. No retries permitted until 2026-09-16 21:21:57.482536316 +0000 UTC m=+4.130948304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs") pod "network-metrics-daemon-x98p4" (UID: "ffed9181-7dab-40e2-aa2a-94f3ef7dae5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:21:56.482646 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.482561    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:21:56.482646 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.482575    2567 projected.go:196] Error preparing data for projected volume kube-api-access-zhswz for pod openshift-network-diagnostics/network-check-target-frkfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:21:56.482646 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.482642    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz podName:d4a729ab-f653-4c7c-a0bb-23569feae959 nodeName:}" failed. No retries permitted until 2026-09-16 21:21:57.482624077 +0000 UTC m=+4.131036046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhswz" (UniqueName: "kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz") pod "network-check-target-frkfh" (UID: "d4a729ab-f653-4c7c-a0bb-23569feae959") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:21:56.816128 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.816048    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 21:16:55 +0000 UTC" deadline="2028-03-14 13:28:12.879298764 +0000 UTC"
Sep 16 21:21:56.816128 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.816130    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13072h6m16.063179431s"
Sep 16 21:21:56.935891 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.935846    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:21:56.936062 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.936012    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:21:56.936460 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.936436    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:56.936595 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:56.936563    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:21:56.947359 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.947328    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kfcgt" event={"ID":"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5","Type":"ContainerStarted","Data":"6bdc575da2a058459bf8e3ace61c2078255424618a9e6327d3c9ebbf396255d1"}
Sep 16 21:21:56.949085 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.949057    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kh5ps" event={"ID":"26787c23-3f06-45d7-bde9-d895e5de9835","Type":"ContainerStarted","Data":"2d5ce2a71bdd64fa5d114816f28f61f4d785e58e04b0cfa58da5935c5d3d75ab"}
Sep 16 21:21:56.951286 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.951262    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr" event={"ID":"5c654f14-60cd-4513-a8cc-9fad85b35314","Type":"ContainerStarted","Data":"5047606befc104c3c50943196ba933b18a50f3ef5a6f3138b15ff5f1e98a0021"}
Sep 16 21:21:56.953432 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.953406    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" event={"ID":"0909f050-da4b-4976-b90b-18d1f4e3f352","Type":"ContainerStarted","Data":"d50e1bcdf6ae6e012db44cc83ed91bd9c18bdddb77f31d15538a38020468ed82"}
Sep 16 21:21:56.955723 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.955701    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vkjfs" event={"ID":"406b94f6-b82c-49f4-b43c-7505af5cee56","Type":"ContainerStarted","Data":"7b70e06f3a12d5a3b4310ce6b34701f0f1674e89f98798eaf4c9d479393f7e5b"}
Sep 16 21:21:56.957344 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.957289    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xd599" event={"ID":"5fb0449b-4fbe-446e-8aa0-9d2c653d440d","Type":"ContainerStarted","Data":"94949115258c177c9ed67bde46b888eb5953e7d07de5090ca1430307f8c61743"}
Sep 16 21:21:56.960359 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.960335    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgq4m" event={"ID":"77e43015-e830-4e21-afed-16ddd27263e9","Type":"ContainerStarted","Data":"fe1da98b5150f038fb8dfc00d7256efebce9d8a7c8ae9201c45a21fe2ddf0577"}
Sep 16 21:21:56.962619 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.962593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f247x" event={"ID":"7386598c-4acd-4326-8f66-f455708aaa53","Type":"ContainerStarted","Data":"fae512b2ae0dd3c0371e67793a3c5d4b6b654104205c2c1ed66bc88f61551cc9"}
Sep 16 21:21:56.964043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:56.964022    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d4fm5" event={"ID":"94aef974-513c-400a-b957-c5b8046ede7d","Type":"ContainerStarted","Data":"0600b7872e31e246e00d209db1c3f47483573825439bb5f3957e77e65be25697"}
Sep 16 21:21:57.266282 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:57.265943    2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 21:21:57.392147 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:57.391526    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:57.392147 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:57.391681    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:21:57.392147 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:57.391749    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret podName:38c2151a-8d75-4772-a0a2-922fb7ca7e20 nodeName:}" failed. No retries permitted until 2026-09-16 21:21:59.391730414 +0000 UTC m=+6.040142398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret") pod "global-pull-secret-syncer-dh4v8" (UID: "38c2151a-8d75-4772-a0a2-922fb7ca7e20") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:21:57.491923 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:57.491869    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:57.492091 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:57.491997    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:21:57.492169 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:57.492143    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:21:57.492169 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:57.492165    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:21:57.492275 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:57.492178    2567 projected.go:196] Error preparing data for projected volume kube-api-access-zhswz for pod openshift-network-diagnostics/network-check-target-frkfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:21:57.492275 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:57.492238    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz podName:d4a729ab-f653-4c7c-a0bb-23569feae959 nodeName:}" failed. No retries permitted until 2026-09-16 21:21:59.492219202 +0000 UTC m=+6.140631183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhswz" (UniqueName: "kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz") pod "network-check-target-frkfh" (UID: "d4a729ab-f653-4c7c-a0bb-23569feae959") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:21:57.492629 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:57.492610    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:21:57.492701 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:57.492663    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs podName:ffed9181-7dab-40e2-aa2a-94f3ef7dae5d nodeName:}" failed. No retries permitted until 2026-09-16 21:21:59.492648887 +0000 UTC m=+6.141060876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs") pod "network-metrics-daemon-x98p4" (UID: "ffed9181-7dab-40e2-aa2a-94f3ef7dae5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:21:57.816504 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:57.816441    2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 21:16:55 +0000 UTC" deadline="2028-02-22 15:28:53.071070721 +0000 UTC"
Sep 16 21:21:57.816504 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:57.816502    2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12570h6m55.254572315s"
Sep 16 21:21:57.935597 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:57.935560    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:57.935772 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:57.935695    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:21:58.936561 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:58.936469    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:21:58.937050 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:58.936614    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:21:58.937050 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:58.937026    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:58.937177 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:58.937132    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:21:59.414574 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:59.414533    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:59.414784 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:59.414772    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:21:59.414846 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:59.414840    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret podName:38c2151a-8d75-4772-a0a2-922fb7ca7e20 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:03.414823535 +0000 UTC m=+10.063235518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret") pod "global-pull-secret-syncer-dh4v8" (UID: "38c2151a-8d75-4772-a0a2-922fb7ca7e20") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:21:59.515140 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:59.515102    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:21:59.515321 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:59.515180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:21:59.515382 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:59.515327    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:21:59.515430 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:59.515396    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs podName:ffed9181-7dab-40e2-aa2a-94f3ef7dae5d nodeName:}" failed. No retries permitted until 2026-09-16 21:22:03.515374894 +0000 UTC m=+10.163786879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs") pod "network-metrics-daemon-x98p4" (UID: "ffed9181-7dab-40e2-aa2a-94f3ef7dae5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:21:59.515820 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:59.515798    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:21:59.515820 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:59.515823    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:21:59.515996 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:59.515836    2567 projected.go:196] Error preparing data for projected volume kube-api-access-zhswz for pod openshift-network-diagnostics/network-check-target-frkfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:21:59.515996 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:59.515880    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz podName:d4a729ab-f653-4c7c-a0bb-23569feae959 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:03.515866247 +0000 UTC m=+10.164278228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhswz" (UniqueName: "kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz") pod "network-check-target-frkfh" (UID: "d4a729ab-f653-4c7c-a0bb-23569feae959") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:21:59.940025 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:21:59.939995    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:21:59.940508 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:21:59.940128    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:00.936466 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:00.936381    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:00.936620 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:00.936573    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:00.936620 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:00.936593    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:00.936744 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:00.936710    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:01.936684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:01.936594    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:01.937134 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:01.937088    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:02.935866 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:02.935830    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:02.936128 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:02.935843    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:02.936128 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:02.936087    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:02.936239 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:02.936195    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:03.453769 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:03.453729    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:03.454201 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:03.453896    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:03.454201 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:03.453978    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret podName:38c2151a-8d75-4772-a0a2-922fb7ca7e20 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:11.453963391 +0000 UTC m=+18.102375373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret") pod "global-pull-secret-syncer-dh4v8" (UID: "38c2151a-8d75-4772-a0a2-922fb7ca7e20") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:03.554622 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:03.554521    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:03.554622 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:03.554595    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:03.554869 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:03.554716    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:03.554869 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:03.554741    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:03.554869 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:03.554807    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs podName:ffed9181-7dab-40e2-aa2a-94f3ef7dae5d nodeName:}" failed. No retries permitted until 2026-09-16 21:22:11.554787254 +0000 UTC m=+18.203199240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs") pod "network-metrics-daemon-x98p4" (UID: "ffed9181-7dab-40e2-aa2a-94f3ef7dae5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:03.554869 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:03.554742    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:03.554869 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:03.554867    2567 projected.go:196] Error preparing data for projected volume kube-api-access-zhswz for pod openshift-network-diagnostics/network-check-target-frkfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:03.555124 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:03.554942    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz podName:d4a729ab-f653-4c7c-a0bb-23569feae959 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:11.5549245 +0000 UTC m=+18.203336481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhswz" (UniqueName: "kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz") pod "network-check-target-frkfh" (UID: "d4a729ab-f653-4c7c-a0bb-23569feae959") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:03.936533 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:03.936498    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:03.936703 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:03.936622    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:04.936704 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:04.936196    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:04.936704 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:04.936326    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:04.936704 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:04.936438    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:04.936704 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:04.936645    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:05.936254 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:05.936223    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:05.936408 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:05.936368    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:06.936340 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:06.936300    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:06.936340 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:06.936329    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:06.936995 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:06.936438    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:06.936995 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:06.936566    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:07.936108 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:07.936074    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:07.936267 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:07.936191    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:08.935982 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:08.935940    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:08.935982 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:08.935967    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:08.936456 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:08.936077    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:08.936456 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:08.936210    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:09.935943 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:09.935896    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:09.936137 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:09.936054    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:10.935947 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:10.935894    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:10.935947 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:10.935925    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:10.936511 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:10.936035    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:10.936511 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:10.936120    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:11.514168 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:11.514116    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:11.514380 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:11.514289    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:11.514380 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:11.514373    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret podName:38c2151a-8d75-4772-a0a2-922fb7ca7e20 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:27.514350853 +0000 UTC m=+34.162762834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret") pod "global-pull-secret-syncer-dh4v8" (UID: "38c2151a-8d75-4772-a0a2-922fb7ca7e20") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:11.614868 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:11.614825    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:11.615041 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:11.614880    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:11.615041 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:11.615014    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:11.615041 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:11.615021    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:11.615187 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:11.615044    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:11.615187 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:11.615058    2567 projected.go:196] Error preparing data for projected volume kube-api-access-zhswz for pod openshift-network-diagnostics/network-check-target-frkfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:11.615187 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:11.615083    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs podName:ffed9181-7dab-40e2-aa2a-94f3ef7dae5d nodeName:}" failed. No retries permitted until 2026-09-16 21:22:27.615064623 +0000 UTC m=+34.263476612 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs") pod "network-metrics-daemon-x98p4" (UID: "ffed9181-7dab-40e2-aa2a-94f3ef7dae5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:11.615187 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:11.615104    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz podName:d4a729ab-f653-4c7c-a0bb-23569feae959 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:27.615092756 +0000 UTC m=+34.263504740 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhswz" (UniqueName: "kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz") pod "network-check-target-frkfh" (UID: "d4a729ab-f653-4c7c-a0bb-23569feae959") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:11.936452 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:11.936371    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:11.936866 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:11.936500    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:12.936231 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:12.936192    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:12.936418 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:12.936347    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:12.936477 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:12.936340    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:12.936477 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:12.936456    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:13.936191 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:13.936158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:13.936352 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:13.936243    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:14.936356 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:14.936117    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:14.937144 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:14.936251    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:14.937144 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:14.936497    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:14.937144 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:14.936665    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:15.010656 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:15.010626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d4fm5" event={"ID":"94aef974-513c-400a-b957-c5b8046ede7d","Type":"ContainerStarted","Data":"af234dec050fa367e8c3d89b3b3b741d085e3ca0bb179deb6b3c219c41afae80"}
Sep 16 21:22:15.012559 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:15.012432    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kh5ps" event={"ID":"26787c23-3f06-45d7-bde9-d895e5de9835","Type":"ContainerStarted","Data":"1fd8960a53a8b9758dfbe1e969b837396edf876d7bc23cecef9b7a6d4bb5a1b6"}
Sep 16 21:22:15.015147 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:15.014954    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" event={"ID":"0909f050-da4b-4976-b90b-18d1f4e3f352","Type":"ContainerStarted","Data":"e383eb40f1b44e3d470e0946d9029458e12725f0a2c5ce0eccc395ea3ec726d1"}
Sep 16 21:22:15.015147 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:15.015042    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" event={"ID":"0909f050-da4b-4976-b90b-18d1f4e3f352","Type":"ContainerStarted","Data":"5f70e74f4c5a27f518a350cccae5307cf41e1c8493b0d751d7ebee313f50aa2c"}
Sep 16 21:22:15.015147 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:15.015094    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" event={"ID":"0909f050-da4b-4976-b90b-18d1f4e3f352","Type":"ContainerStarted","Data":"1df36c12c86b85648b3b1202696c6124a2c3a6b45507b9ebaf5c4dbc587c1414"}
Sep 16 21:22:15.016555 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:15.016504    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal" event={"ID":"9292673e818dc6000ad469c880116e19","Type":"ContainerStarted","Data":"8003ece22a4c4a72f38bf29283c00e17d1225d2c40ae5e87d123585794006e95"}
Sep 16 21:22:15.027723 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:15.027494    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d4fm5" podStartSLOduration=2.802936853 podStartE2EDuration="21.027480958s" totalImagesPullingTime="18.224544105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:15.027008237 +0000 UTC m=+21.675420226" watchObservedRunningTime="2026-09-16 21:22:15.027480958 +0000 UTC m=+21.675892947"
Sep 16 21:22:15.038884 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:15.038833    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-209.ec2.internal" podStartSLOduration=21.03881439 podStartE2EDuration="21.03881439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:15.0385057 +0000 UTC m=+21.686917690" watchObservedRunningTime="2026-09-16 21:22:15.03881439 +0000 UTC m=+21.687226378"
Sep 16 21:22:15.936815 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:15.936561    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:15.937617 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:15.936843    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:16.020837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.020802    2567 generic.go:356] "Generic (PLEG): container finished" podID="2885e0b22b161a32ae54bba2bc4adfb1" containerID="bf0fd75fe9f0941096e072e3072edc8d9683ba565d1d9e3040573e43415963b4" exitCode=0
Sep 16 21:22:16.021024 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.020881    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal" event={"ID":"2885e0b22b161a32ae54bba2bc4adfb1","Type":"ContainerDied","Data":"bf0fd75fe9f0941096e072e3072edc8d9683ba565d1d9e3040573e43415963b4"}
Sep 16 21:22:16.022634 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.022593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f247x" event={"ID":"7386598c-4acd-4326-8f66-f455708aaa53","Type":"ContainerStarted","Data":"17e93a14ac093ba22c292e72540ca2b14654f1767149c689ea001b08a8f49078"}
Sep 16 21:22:16.024314 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.024292    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kfcgt" event={"ID":"b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5","Type":"ContainerStarted","Data":"7874fa8a3d5292924b5832705eb914157889086e85e314032d3c4cb792b667ab"}
Sep 16 21:22:16.025942 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.025920    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr" event={"ID":"5c654f14-60cd-4513-a8cc-9fad85b35314","Type":"ContainerStarted","Data":"67b6417d6cfe4eb8447d9da92d79b0340c724d4aa02a09057c026ce67ed001d6"}
Sep 16 21:22:16.029219 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.029200    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" event={"ID":"0909f050-da4b-4976-b90b-18d1f4e3f352","Type":"ContainerStarted","Data":"03796a66adc2fb4a723fb4722620d937ae500b5869f18c42d5bef1a92080de6f"}
Sep 16 21:22:16.029326 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.029255    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" event={"ID":"0909f050-da4b-4976-b90b-18d1f4e3f352","Type":"ContainerStarted","Data":"60c76b039263acc2e1317c84f254345898e076bbf6838d6dfe21dbb06562e262"}
Sep 16 21:22:16.029326 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.029299    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" event={"ID":"0909f050-da4b-4976-b90b-18d1f4e3f352","Type":"ContainerStarted","Data":"d399e6ccd4e95e9b00b606ed1de73b922a3b1b304f3c8a71dd866f32f48276e4"}
Sep 16 21:22:16.030814 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.030792    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vkjfs" event={"ID":"406b94f6-b82c-49f4-b43c-7505af5cee56","Type":"ContainerStarted","Data":"81b7b3bd941cdfa10c2d469d095c73a7dfb8806e31b320315977aba4d6f79a3f"}
Sep 16 21:22:16.032547 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.032503    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xd599" event={"ID":"5fb0449b-4fbe-446e-8aa0-9d2c653d440d","Type":"ContainerStarted","Data":"cdc0685f716bf1f0c5d0a2778c4d2c5010fdc2fcb51ecd620c74ec5989089204"}
Sep 16 21:22:16.034251 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.034231    2567 generic.go:356] "Generic (PLEG): container finished" podID="77e43015-e830-4e21-afed-16ddd27263e9" containerID="e0f8b382c9195bd76474bb2e0a0eb01926e1034cef07fde2f2710f957d46b579" exitCode=0
Sep 16 21:22:16.034350 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.034326    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgq4m" event={"ID":"77e43015-e830-4e21-afed-16ddd27263e9","Type":"ContainerDied","Data":"e0f8b382c9195bd76474bb2e0a0eb01926e1034cef07fde2f2710f957d46b579"}
Sep 16 21:22:16.035700 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.035657    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kh5ps" podStartSLOduration=3.817598264 podStartE2EDuration="22.035640708s" totalImagesPullingTime="18.218042444s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:15.053718742 +0000 UTC m=+21.702130731" watchObservedRunningTime="2026-09-16 21:22:16.035640708 +0000 UTC m=+22.684052696"
Sep 16 21:22:16.120156 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.120096    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xd599" podStartSLOduration=3.9312673460000003 podStartE2EDuration="22.120072399s" totalImagesPullingTime="18.188805053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:16.101771639 +0000 UTC m=+22.750183632" watchObservedRunningTime="2026-09-16 21:22:16.120072399 +0000 UTC m=+22.768484390"
Sep 16 21:22:16.120367 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.120292    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vkjfs" podStartSLOduration=3.919926242 podStartE2EDuration="22.120280574s" totalImagesPullingTime="18.200354332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:16.120245051 +0000 UTC m=+22.768657040" watchObservedRunningTime="2026-09-16 21:22:16.120280574 +0000 UTC m=+22.768692563"
Sep 16 21:22:16.133850 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.133786    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kfcgt" podStartSLOduration=4.363380277 podStartE2EDuration="22.133767622s" totalImagesPullingTime="17.770387345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:16.133559824 +0000 UTC m=+22.781971813" watchObservedRunningTime="2026-09-16 21:22:16.133767622 +0000 UTC m=+22.782179610"
Sep 16 21:22:16.147249 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.147202    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f247x" podStartSLOduration=8.087357157 podStartE2EDuration="22.147187737s" totalImagesPullingTime="14.05983058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:16.146499373 +0000 UTC m=+22.794911361" watchObservedRunningTime="2026-09-16 21:22:16.147187737 +0000 UTC m=+22.795599725"
Sep 16 21:22:16.289474 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.289438    2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 21:22:16.935734 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.935704    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:16.935961 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.935703    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:16.935961 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:16.935832    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:16.935961 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:16.935912    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:16.968756 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.968585    2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T21:22:16.289468004Z","UUID":"77571882-c515-44ab-82d7-8cd1058f52cd","Handler":null,"Name":"","Endpoint":""}
Sep 16 21:22:16.970147 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.970123    2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 21:22:16.970267 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:16.970160    2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 21:22:17.040182 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:17.040142    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal" event={"ID":"2885e0b22b161a32ae54bba2bc4adfb1","Type":"ContainerStarted","Data":"3aa56aace4d186043cc3629aafb4944552844ef79982f980a2a29d22b06cffd5"}
Sep 16 21:22:17.042969 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:17.042938    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr" event={"ID":"5c654f14-60cd-4513-a8cc-9fad85b35314","Type":"ContainerStarted","Data":"de08137731c2302d04d46068aef497c915aacf90c5c0b95aba19ec331a18bcd9"}
Sep 16 21:22:17.053068 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:17.053003    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-209.ec2.internal" podStartSLOduration=23.052982093 podStartE2EDuration="23.052982093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:17.052958444 +0000 UTC m=+23.701370433" watchObservedRunningTime="2026-09-16 21:22:17.052982093 +0000 UTC m=+23.701394059"
Sep 16 21:22:17.477940 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:17.477585    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:22:17.478301 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:17.478278    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:22:17.936409 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:17.936360    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:17.936585 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:17.936519    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:18.047749 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:18.047712    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr" event={"ID":"5c654f14-60cd-4513-a8cc-9fad85b35314","Type":"ContainerStarted","Data":"663d52706f8272624d2aeb0c8035b30ef9ae26c2e924700def43b1bc11ada35b"}
Sep 16 21:22:18.051676 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:18.051648    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" event={"ID":"0909f050-da4b-4976-b90b-18d1f4e3f352","Type":"ContainerStarted","Data":"cf9398c5aa3f8ad2d360ec052e7619eb609f122e9f4be3ede81f70f40cc436dd"}
Sep 16 21:22:18.052119 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:18.052095    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:22:18.052593 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:18.052573    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f247x"
Sep 16 21:22:18.065468 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:18.065418    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-22tnr" podStartSLOduration=3.815657865 podStartE2EDuration="24.065403057s" totalImagesPullingTime="20.249745192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:18.064030093 +0000 UTC m=+24.712442080" watchObservedRunningTime="2026-09-16 21:22:18.065403057 +0000 UTC m=+24.713815045"
Sep 16 21:22:18.935820 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:18.935781    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:18.935820 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:18.935818    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:18.936146 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:18.935931    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:18.936146 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:18.936031    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:19.939320 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:19.939292    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:19.939741 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:19.939380    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:20.060861 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:20.060821    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" event={"ID":"0909f050-da4b-4976-b90b-18d1f4e3f352","Type":"ContainerStarted","Data":"17d3c53d4b58316b9a91ce39edccbb3a082ee8580bedadb81a97efc29c5f43a1"}
Sep 16 21:22:20.088440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:20.088366    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-579wp" podStartSLOduration=7.449233533 podStartE2EDuration="26.088343207s" totalImagesPullingTime="18.639109674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:20.087268248 +0000 UTC m=+26.735680237" watchObservedRunningTime="2026-09-16 21:22:20.088343207 +0000 UTC m=+26.736755198"
Sep 16 21:22:20.936041 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:20.935806    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:20.936230 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:20.935805    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:20.936230 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:20.936170    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:20.936346 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:20.936223    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:21.063847 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.063815    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:22:21.064307 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.063921    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:22:21.064307 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.063973    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:22:21.080750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.080711    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:22:21.081287 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.081263    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:22:21.758301 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.758271    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dh4v8"]
Sep 16 21:22:21.758503 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.758415    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:21.758556 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:21.758506    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:21.761138 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.761110    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x98p4"]
Sep 16 21:22:21.761312 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.761214    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:21.761312 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:21.761294    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:21.779823 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.779794    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-frkfh"]
Sep 16 21:22:21.780008 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:21.779925    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:21.780062 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:21.780009    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:23.069240 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:23.069195    2567 generic.go:356] "Generic (PLEG): container finished" podID="77e43015-e830-4e21-afed-16ddd27263e9" containerID="8e3d6adfddfaa83a7a9bb7ef66bae8c112041eb39be8dead5c048f5cf6f9c177" exitCode=0
Sep 16 21:22:23.069882 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:23.069286    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgq4m" event={"ID":"77e43015-e830-4e21-afed-16ddd27263e9","Type":"ContainerDied","Data":"8e3d6adfddfaa83a7a9bb7ef66bae8c112041eb39be8dead5c048f5cf6f9c177"}
Sep 16 21:22:23.939700 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:23.939622    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:23.939700 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:23.939657    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:23.939917 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:23.939753    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:23.939917 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:23.939763    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:23.939917 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:23.939843    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:23.939917 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:23.939886    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:25.076789 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:25.076754    2567 generic.go:356] "Generic (PLEG): container finished" podID="77e43015-e830-4e21-afed-16ddd27263e9" containerID="4eca9f06c8571ae2329c073c4afc8877870ed5e211ccb82018feeceda950e147" exitCode=0
Sep 16 21:22:25.076789 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:25.076794    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgq4m" event={"ID":"77e43015-e830-4e21-afed-16ddd27263e9","Type":"ContainerDied","Data":"4eca9f06c8571ae2329c073c4afc8877870ed5e211ccb82018feeceda950e147"}
Sep 16 21:22:25.936519 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:25.936482    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:25.936718 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:25.936482    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:25.936718 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:25.936609    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dh4v8" podUID="38c2151a-8d75-4772-a0a2-922fb7ca7e20"
Sep 16 21:22:25.936718 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:25.936487    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:25.936718 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:25.936676    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x98p4" podUID="ffed9181-7dab-40e2-aa2a-94f3ef7dae5d"
Sep 16 21:22:25.936895 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:25.936769    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-frkfh" podUID="d4a729ab-f653-4c7c-a0bb-23569feae959"
Sep 16 21:22:27.084692 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.084661    2567 generic.go:356] "Generic (PLEG): container finished" podID="77e43015-e830-4e21-afed-16ddd27263e9" containerID="04ad3c018bf30cd7dc43e9a976f15ba0018bacfd26e8497ad39291aa69b75913" exitCode=0
Sep 16 21:22:27.085087 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.084722    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgq4m" event={"ID":"77e43015-e830-4e21-afed-16ddd27263e9","Type":"ContainerDied","Data":"04ad3c018bf30cd7dc43e9a976f15ba0018bacfd26e8497ad39291aa69b75913"}
Sep 16 21:22:27.518944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.518894    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:27.519068 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.519049    2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:27.519123 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.519114    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret podName:38c2151a-8d75-4772-a0a2-922fb7ca7e20 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:59.519098694 +0000 UTC m=+66.167510659 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret") pod "global-pull-secret-syncer-dh4v8" (UID: "38c2151a-8d75-4772-a0a2-922fb7ca7e20") : object "kube-system"/"original-pull-secret" not registered
Sep 16 21:22:27.619914 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.619877    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:27.620075 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.619957    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:27.620075 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.620025    2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:27.620075 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.620059    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 21:22:27.620075 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.620076    2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 21:22:27.620195 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.620087    2567 projected.go:196] Error preparing data for projected volume kube-api-access-zhswz for pod openshift-network-diagnostics/network-check-target-frkfh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:27.620195 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.620104    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs podName:ffed9181-7dab-40e2-aa2a-94f3ef7dae5d nodeName:}" failed. No retries permitted until 2026-09-16 21:22:59.620086427 +0000 UTC m=+66.268498392 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs") pod "network-metrics-daemon-x98p4" (UID: "ffed9181-7dab-40e2-aa2a-94f3ef7dae5d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 21:22:27.620195 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.620125    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz podName:d4a729ab-f653-4c7c-a0bb-23569feae959 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:59.620114715 +0000 UTC m=+66.268526694 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhswz" (UniqueName: "kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz") pod "network-check-target-frkfh" (UID: "d4a729ab-f653-4c7c-a0bb-23569feae959") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 21:22:27.629101 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.629075    2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-209.ec2.internal" event="NodeReady"
Sep 16 21:22:27.629210 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.629205    2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 21:22:27.678248 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.678215    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f97687479-dkdrq"]
Sep 16 21:22:27.681510 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.681484    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5f57db6868-lgg22"]
Sep 16 21:22:27.681824 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.681791    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.685052 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.685032    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"]
Sep 16 21:22:27.685162 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.685149    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.687017 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.686994    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 21:22:27.688150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.688108    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-s2szh"]
Sep 16 21:22:27.688150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.688108    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 21:22:27.688336 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.688161    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 21:22:27.688336 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.688194    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:27.688336 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.688220    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5496m\""
Sep 16 21:22:27.689294 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.689267    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.689294 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.689282    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 16 21:22:27.690055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.690038    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ckjjz\""
Sep 16 21:22:27.690453 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.690438    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.690794 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.690774    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn"]
Sep 16 21:22:27.690931 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.690884    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.691403 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.691380    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 16 21:22:27.691494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.691445    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 21:22:27.691494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.691456    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 16 21:22:27.692798 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.692783    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 16 21:22:27.694026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.694007    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wmdmb\""
Sep 16 21:22:27.694529 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.694395    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.694529 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.694407    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.694648 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.694573    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 21:22:27.694685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.694671    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn"
Sep 16 21:22:27.694891 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.694871    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 16 21:22:27.695078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.694934    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 16 21:22:27.695078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.694942    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.695206 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.695166    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.695340 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.695325    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vbwq7\""
Sep 16 21:22:27.699618 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.699601    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 21:22:27.701570 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.701548    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 16 21:22:27.702568 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.702549    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.702807 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.702566    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.706357 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.706335    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-psmpl\""
Sep 16 21:22:27.707664 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.707644    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f97687479-dkdrq"]
Sep 16 21:22:27.709583 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.709564    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn"]
Sep 16 21:22:27.713931 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.713851    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-s2szh"]
Sep 16 21:22:27.715414 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.715378    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-45xsc"]
Sep 16 21:22:27.720163 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.720143    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"]
Sep 16 21:22:27.720293 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.720271    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.722756 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.722734    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 21:22:27.722872 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.722781    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rcnph\""
Sep 16 21:22:27.722872 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.722866    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 21:22:27.723927 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.723892    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"]
Sep 16 21:22:27.724026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.724008    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:27.726875 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.726852    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.727008 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.726991    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7zskn\""
Sep 16 21:22:27.727302 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.727287    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.727541 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.727524    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"]
Sep 16 21:22:27.727632 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.727616    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 16 21:22:27.727688 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.727634    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.731053 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.731035    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.731147 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.731074    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq"]
Sep 16 21:22:27.731207 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.731188    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.731413 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.731393    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 16 21:22:27.731658 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.731639    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 16 21:22:27.732078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.732062    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lhmq7\""
Sep 16 21:22:27.732499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.732481    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.735006 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.734955    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8p9nq"]
Sep 16 21:22:27.735258 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.735240    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq"
Sep 16 21:22:27.737852 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.737825    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 21:22:27.738053 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.738034    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.738319 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.738300    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f57db6868-lgg22"]
Sep 16 21:22:27.738412 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.738335    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jhhbw"]
Sep 16 21:22:27.738412 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.738391    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.739043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.738872    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.739043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.738889    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kcshj\""
Sep 16 21:22:27.739043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.738923    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.739043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.738937    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.739043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.738944    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xxcpz\""
Sep 16 21:22:27.739339 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.739324    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 21:22:27.741342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.741266    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 21:22:27.741410 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.741397    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-79tsb\""
Sep 16 21:22:27.741732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.741714    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.741818 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.741715    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.743098 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.743078    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 21:22:27.743200 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.743183    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhhbw"
Sep 16 21:22:27.748236 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.748202    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-77lfh\""
Sep 16 21:22:27.748520 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.748483    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 21:22:27.749225 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.749204    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 21:22:27.749459 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.749442    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 21:22:27.750323 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.750300    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"]
Sep 16 21:22:27.758930 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.758889    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8p9nq"]
Sep 16 21:22:27.759766 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.759745    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"]
Sep 16 21:22:27.760699 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.760681    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45xsc"]
Sep 16 21:22:27.761380 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.761356    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhhbw"]
Sep 16 21:22:27.763449 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.763315    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"]
Sep 16 21:22:27.767647 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.767376    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 21:22:27.770729 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.770707    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq"]
Sep 16 21:22:27.774276 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.774255    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"]
Sep 16 21:22:27.821229 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821192    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f518039-df50-4fbb-9d28-f6e554a1a784-trusted-ca-bundle\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.821229 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2908a138-2d0e-4eb8-8b35-742e861fb491-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g9pqs\" (UID: \"2908a138-2d0e-4eb8-8b35-742e861fb491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.821440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821249    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:27.821440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821300    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjdf\" (UniqueName: \"kubernetes.io/projected/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-kube-api-access-rkjdf\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:27.821440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-trusted-ca\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.821440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2908a138-2d0e-4eb8-8b35-742e861fb491-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g9pqs\" (UID: \"2908a138-2d0e-4eb8-8b35-742e861fb491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.821440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f518039-df50-4fbb-9d28-f6e554a1a784-serving-cert\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.821440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821403    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.821440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821420    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7qsc\" (UniqueName: \"kubernetes.io/projected/9166aeb5-51d3-4a3e-8dbb-c6ff8b46ac1a-kube-api-access-n7qsc\") pod \"volume-data-source-validator-86695b7cc7-skqzn\" (UID: \"9166aeb5-51d3-4a3e-8dbb-c6ff8b46ac1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn"
Sep 16 21:22:27.821440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821436    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f518039-df50-4fbb-9d28-f6e554a1a784-tmp\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c2c6551-1309-467a-8886-a3c278d89c03-cert\") pod \"ingress-canary-jhhbw\" (UID: \"5c2c6551-1309-467a-8886-a3c278d89c03\") " pod="openshift-ingress-canary/ingress-canary-jhhbw"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821465    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khpfn\" (UniqueName: \"kubernetes.io/projected/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-kube-api-access-khpfn\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821488    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-bound-sa-token\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821548    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c2b47d0-5a18-4998-824f-e8d425e414c3-ca-trust-extracted\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821568    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8049ff-a69e-4607-b24f-b98b40f53fc2-config-volume\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821588    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-certificates\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821606    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-default-certificate\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821619    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821635    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-image-registry-private-configuration\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821651    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5258e8e0-274b-4247-966f-99e56319eb18-config\") pod \"service-ca-operator-cf7c776b7-hf4dv\" (UID: \"5258e8e0-274b-4247-966f-99e56319eb18\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.821672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821669    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-serving-cert\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821686    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfb7j\" (UniqueName: \"kubernetes.io/projected/77d6ae3a-7648-4389-9173-9b6487ae023d-kube-api-access-gfb7j\") pod \"network-check-source-6f8b768455-c4ckq\" (UID: \"77d6ae3a-7648-4389-9173-9b6487ae023d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821702    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8c2q\" (UniqueName: \"kubernetes.io/projected/5c2c6551-1309-467a-8886-a3c278d89c03-kube-api-access-n8c2q\") pod \"ingress-canary-jhhbw\" (UID: \"5c2c6551-1309-467a-8886-a3c278d89c03\") " pod="openshift-ingress-canary/ingress-canary-jhhbw"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821719    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xh5\" (UniqueName: \"kubernetes.io/projected/863ab98a-8183-4e24-aba7-8d57e12368c6-kube-api-access-92xh5\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821740    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f518039-df50-4fbb-9d28-f6e554a1a784-service-ca-bundle\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821767    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-config\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821789    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-trusted-ca\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821823    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8049ff-a69e-4607-b24f-b98b40f53fc2-tmp-dir\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821839    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f518039-df50-4fbb-9d28-f6e554a1a784-snapshots\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821853    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-stats-auth\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821868    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bswbg\" (UniqueName: \"kubernetes.io/projected/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-kube-api-access-bswbg\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821884    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821925    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821949    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821966    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-installation-pull-secrets\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.822057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.821982    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5258e8e0-274b-4247-966f-99e56319eb18-serving-cert\") pod \"service-ca-operator-cf7c776b7-hf4dv\" (UID: \"5258e8e0-274b-4247-966f-99e56319eb18\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.822466 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.822025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t867z\" (UniqueName: \"kubernetes.io/projected/1e8049ff-a69e-4607-b24f-b98b40f53fc2-kube-api-access-t867z\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.822466 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.822043    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztzt\" (UniqueName: \"kubernetes.io/projected/9f518039-df50-4fbb-9d28-f6e554a1a784-kube-api-access-xztzt\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.822466 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.822066    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8nd\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-kube-api-access-qj8nd\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.822466 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.822086    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7547j\" (UniqueName: \"kubernetes.io/projected/5258e8e0-274b-4247-966f-99e56319eb18-kube-api-access-7547j\") pod \"service-ca-operator-cf7c776b7-hf4dv\" (UID: \"5258e8e0-274b-4247-966f-99e56319eb18\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.822466 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.822124    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwjs7\" (UniqueName: \"kubernetes.io/projected/2908a138-2d0e-4eb8-8b35-742e861fb491-kube-api-access-bwjs7\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g9pqs\" (UID: \"2908a138-2d0e-4eb8-8b35-742e861fb491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.822466 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.822140    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:27.923233 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923200    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-certificates\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.923233 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923231    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-default-certificate\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.923440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923247    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.923440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923415    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-image-registry-private-configuration\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.923440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923439    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5258e8e0-274b-4247-966f-99e56319eb18-config\") pod \"service-ca-operator-cf7c776b7-hf4dv\" (UID: \"5258e8e0-274b-4247-966f-99e56319eb18\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.923578 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923456    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-serving-cert\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.923578 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.923527    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:28.423505699 +0000 UTC m=+35.071917678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : configmap references non-existent config key: service-ca.crt
Sep 16 21:22:27.923578 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923567    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfb7j\" (UniqueName: \"kubernetes.io/projected/77d6ae3a-7648-4389-9173-9b6487ae023d-kube-api-access-gfb7j\") pod \"network-check-source-6f8b768455-c4ckq\" (UID: \"77d6ae3a-7648-4389-9173-9b6487ae023d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq"
Sep 16 21:22:27.923734 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8c2q\" (UniqueName: \"kubernetes.io/projected/5c2c6551-1309-467a-8886-a3c278d89c03-kube-api-access-n8c2q\") pod \"ingress-canary-jhhbw\" (UID: \"5c2c6551-1309-467a-8886-a3c278d89c03\") " pod="openshift-ingress-canary/ingress-canary-jhhbw"
Sep 16 21:22:27.923734 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923632    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xh5\" (UniqueName: \"kubernetes.io/projected/863ab98a-8183-4e24-aba7-8d57e12368c6-kube-api-access-92xh5\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.923734 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923656    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f518039-df50-4fbb-9d28-f6e554a1a784-service-ca-bundle\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.923734 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923696    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-config\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.923941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923742    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-trusted-ca\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.923941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923766    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8049ff-a69e-4607-b24f-b98b40f53fc2-tmp-dir\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.923941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923790    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f518039-df50-4fbb-9d28-f6e554a1a784-snapshots\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.923941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923820    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-stats-auth\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.923941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923849    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bswbg\" (UniqueName: \"kubernetes.io/projected/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-kube-api-access-bswbg\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:27.924203 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.923981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-certificates\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.924203 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924119    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5258e8e0-274b-4247-966f-99e56319eb18-config\") pod \"service-ca-operator-cf7c776b7-hf4dv\" (UID: \"5258e8e0-274b-4247-966f-99e56319eb18\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924335    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924374    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924407    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924432    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-installation-pull-secrets\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924455    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5258e8e0-274b-4247-966f-99e56319eb18-serving-cert\") pod \"service-ca-operator-cf7c776b7-hf4dv\" (UID: \"5258e8e0-274b-4247-966f-99e56319eb18\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924498    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t867z\" (UniqueName: \"kubernetes.io/projected/1e8049ff-a69e-4607-b24f-b98b40f53fc2-kube-api-access-t867z\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924524    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztzt\" (UniqueName: \"kubernetes.io/projected/9f518039-df50-4fbb-9d28-f6e554a1a784-kube-api-access-xztzt\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924527    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f518039-df50-4fbb-9d28-f6e554a1a784-service-ca-bundle\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8nd\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-kube-api-access-qj8nd\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924602    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7547j\" (UniqueName: \"kubernetes.io/projected/5258e8e0-274b-4247-966f-99e56319eb18-kube-api-access-7547j\") pod \"service-ca-operator-cf7c776b7-hf4dv\" (UID: \"5258e8e0-274b-4247-966f-99e56319eb18\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924613    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e8049ff-a69e-4607-b24f-b98b40f53fc2-tmp-dir\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924655    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwjs7\" (UniqueName: \"kubernetes.io/projected/2908a138-2d0e-4eb8-8b35-742e861fb491-kube-api-access-bwjs7\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g9pqs\" (UID: \"2908a138-2d0e-4eb8-8b35-742e861fb491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924680    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.924542    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-config\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.924944 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.924801    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925034    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9f518039-df50-4fbb-9d28-f6e554a1a784-snapshots\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925163    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-trusted-ca\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.925300    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.925316    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f97687479-dkdrq: secret "image-registry-tls" not found
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925325    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f518039-df50-4fbb-9d28-f6e554a1a784-trusted-ca-bundle\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925358    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2908a138-2d0e-4eb8-8b35-742e861fb491-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g9pqs\" (UID: \"2908a138-2d0e-4eb8-8b35-742e861fb491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.925371    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls podName:9c2b47d0-5a18-4998-824f-e8d425e414c3 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:28.425354068 +0000 UTC m=+35.073766049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls") pod "image-registry-5f97687479-dkdrq" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3") : secret "image-registry-tls" not found
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.925460    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.925503    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:28.42549066 +0000 UTC m=+35.073902637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : secret "router-metrics-certs-default" not found
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925522    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925525    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.925543    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls podName:2cfc325f-85d2-4a82-aa0e-12c2ff6beb14 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:28.425534063 +0000 UTC m=+35.073946035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5ldzz" (UID: "2cfc325f-85d2-4a82-aa0e-12c2ff6beb14") : secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:27.925719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925720    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjdf\" (UniqueName: \"kubernetes.io/projected/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-kube-api-access-rkjdf\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925760    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-trusted-ca\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925798    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2908a138-2d0e-4eb8-8b35-742e861fb491-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g9pqs\" (UID: \"2908a138-2d0e-4eb8-8b35-742e861fb491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925832    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f518039-df50-4fbb-9d28-f6e554a1a784-serving-cert\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925860    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925885    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7qsc\" (UniqueName: \"kubernetes.io/projected/9166aeb5-51d3-4a3e-8dbb-c6ff8b46ac1a-kube-api-access-n7qsc\") pod \"volume-data-source-validator-86695b7cc7-skqzn\" (UID: \"9166aeb5-51d3-4a3e-8dbb-c6ff8b46ac1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925933    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f518039-df50-4fbb-9d28-f6e554a1a784-tmp\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925958    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c2c6551-1309-467a-8886-a3c278d89c03-cert\") pod \"ingress-canary-jhhbw\" (UID: \"5c2c6551-1309-467a-8886-a3c278d89c03\") " pod="openshift-ingress-canary/ingress-canary-jhhbw"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.925987    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khpfn\" (UniqueName: \"kubernetes.io/projected/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-kube-api-access-khpfn\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.926018    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-bound-sa-token\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.926058    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c2b47d0-5a18-4998-824f-e8d425e414c3-ca-trust-extracted\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.926085    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8049ff-a69e-4607-b24f-b98b40f53fc2-config-volume\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.926184    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2908a138-2d0e-4eb8-8b35-742e861fb491-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g9pqs\" (UID: \"2908a138-2d0e-4eb8-8b35-742e861fb491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.926295    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.926339    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls podName:f980eff7-4c6c-43da-b2e6-f62f01bda7ee nodeName:}" failed. No retries permitted until 2026-09-16 21:22:28.426325439 +0000 UTC m=+35.074737418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rb794" (UID: "f980eff7-4c6c-43da-b2e6-f62f01bda7ee") : secret "samples-operator-tls" not found
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.926506    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f518039-df50-4fbb-9d28-f6e554a1a784-trusted-ca-bundle\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.927348 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.926595    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 21:22:27.928181 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:27.926650    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls podName:1e8049ff-a69e-4607-b24f-b98b40f53fc2 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:28.426632966 +0000 UTC m=+35.075044943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls") pod "dns-default-45xsc" (UID: "1e8049ff-a69e-4607-b24f-b98b40f53fc2") : secret "dns-default-metrics-tls" not found
Sep 16 21:22:27.928181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.926693    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f518039-df50-4fbb-9d28-f6e554a1a784-tmp\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.928181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.926982    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e8049ff-a69e-4607-b24f-b98b40f53fc2-config-volume\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.928181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.927137    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c2b47d0-5a18-4998-824f-e8d425e414c3-ca-trust-extracted\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.928181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.927407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-trusted-ca\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.929157 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.928847    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-image-registry-private-configuration\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.929385 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.929193    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9f518039-df50-4fbb-9d28-f6e554a1a784-serving-cert\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:27.929385 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.929213    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-serving-cert\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.929385 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.929257    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-installation-pull-secrets\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.929520 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.929427    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-stats-auth\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.929520 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.929460    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2908a138-2d0e-4eb8-8b35-742e861fb491-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g9pqs\" (UID: \"2908a138-2d0e-4eb8-8b35-742e861fb491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.929604 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.929582    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5258e8e0-274b-4247-966f-99e56319eb18-serving-cert\") pod \"service-ca-operator-cf7c776b7-hf4dv\" (UID: \"5258e8e0-274b-4247-966f-99e56319eb18\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.929866 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.929850    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-default-certificate\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.930437 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.930415    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c2c6551-1309-467a-8886-a3c278d89c03-cert\") pod \"ingress-canary-jhhbw\" (UID: \"5c2c6551-1309-467a-8886-a3c278d89c03\") " pod="openshift-ingress-canary/ingress-canary-jhhbw"
Sep 16 21:22:27.936364 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.936334    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:27.936364 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.936354    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:27.936509 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.936488    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:27.936870 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.936731    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8c2q\" (UniqueName: \"kubernetes.io/projected/5c2c6551-1309-467a-8886-a3c278d89c03-kube-api-access-n8c2q\") pod \"ingress-canary-jhhbw\" (UID: \"5c2c6551-1309-467a-8886-a3c278d89c03\") " pod="openshift-ingress-canary/ingress-canary-jhhbw"
Sep 16 21:22:27.937226 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.937178    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7qsc\" (UniqueName: \"kubernetes.io/projected/9166aeb5-51d3-4a3e-8dbb-c6ff8b46ac1a-kube-api-access-n7qsc\") pod \"volume-data-source-validator-86695b7cc7-skqzn\" (UID: \"9166aeb5-51d3-4a3e-8dbb-c6ff8b46ac1a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn"
Sep 16 21:22:27.940807 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.940758    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8nd\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-kube-api-access-qj8nd\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.941836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.941091    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdj86\""
Sep 16 21:22:27.941836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.941135    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 21:22:27.941836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.941206    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t867z\" (UniqueName: \"kubernetes.io/projected/1e8049ff-a69e-4607-b24f-b98b40f53fc2-kube-api-access-t867z\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:27.941836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.941536    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfb7j\" (UniqueName: \"kubernetes.io/projected/77d6ae3a-7648-4389-9173-9b6487ae023d-kube-api-access-gfb7j\") pod \"network-check-source-6f8b768455-c4ckq\" (UID: \"77d6ae3a-7648-4389-9173-9b6487ae023d\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq"
Sep 16 21:22:27.941836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.941548    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8xpmt\""
Sep 16 21:22:27.941836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.941620    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 21:22:27.941836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.941725    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwjs7\" (UniqueName: \"kubernetes.io/projected/2908a138-2d0e-4eb8-8b35-742e861fb491-kube-api-access-bwjs7\") pod \"kube-storage-version-migrator-operator-6c8f84f897-g9pqs\" (UID: \"2908a138-2d0e-4eb8-8b35-742e861fb491\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:27.941836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.941774    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xh5\" (UniqueName: \"kubernetes.io/projected/863ab98a-8183-4e24-aba7-8d57e12368c6-kube-api-access-92xh5\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:27.943510 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.943484    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7547j\" (UniqueName: \"kubernetes.io/projected/5258e8e0-274b-4247-966f-99e56319eb18-kube-api-access-7547j\") pod \"service-ca-operator-cf7c776b7-hf4dv\" (UID: \"5258e8e0-274b-4247-966f-99e56319eb18\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:27.943885 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.943865    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-bound-sa-token\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:27.944050 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.944028    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bswbg\" (UniqueName: \"kubernetes.io/projected/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-kube-api-access-bswbg\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:27.944793 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.944767    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khpfn\" (UniqueName: \"kubernetes.io/projected/f2378ae2-b1f3-41b3-a99c-1aa20b008f68-kube-api-access-khpfn\") pod \"console-operator-696884fb9c-8p9nq\" (UID: \"f2378ae2-b1f3-41b3-a99c-1aa20b008f68\") " pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:27.945007 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.944987    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjdf\" (UniqueName: \"kubernetes.io/projected/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-kube-api-access-rkjdf\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:27.945327 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:27.945307    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztzt\" (UniqueName: \"kubernetes.io/projected/9f518039-df50-4fbb-9d28-f6e554a1a784-kube-api-access-xztzt\") pod \"insights-operator-5844b856db-s2szh\" (UID: \"9f518039-df50-4fbb-9d28-f6e554a1a784\") " pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:28.016726 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.016649    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-s2szh"
Sep 16 21:22:28.022986 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.022965    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn"
Sep 16 21:22:28.044507 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.044468    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"
Sep 16 21:22:28.052553 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.052021    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"
Sep 16 21:22:28.060303 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.060279    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq"
Sep 16 21:22:28.069244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.068999    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:28.074258 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.074083    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhhbw"
Sep 16 21:22:28.315609 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.315303    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-s2szh"]
Sep 16 21:22:28.318502 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.318467    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn"]
Sep 16 21:22:28.320569 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:28.320529    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f518039_df50_4fbb_9d28_f6e554a1a784.slice/crio-54ce67a636f58077eaa3f4b6389f389643a5acf69d6186847f74317122c8cdda WatchSource:0}: Error finding container 54ce67a636f58077eaa3f4b6389f389643a5acf69d6186847f74317122c8cdda: Status 404 returned error can't find the container with id 54ce67a636f58077eaa3f4b6389f389643a5acf69d6186847f74317122c8cdda
Sep 16 21:22:28.321830 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:28.321797    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9166aeb5_51d3_4a3e_8dbb_c6ff8b46ac1a.slice/crio-76f07b791447ed7cfbdaf3eb2572fe3687327bd24d6a437d8de8047a298fb06e WatchSource:0}: Error finding container 76f07b791447ed7cfbdaf3eb2572fe3687327bd24d6a437d8de8047a298fb06e: Status 404 returned error can't find the container with id 76f07b791447ed7cfbdaf3eb2572fe3687327bd24d6a437d8de8047a298fb06e
Sep 16 21:22:28.373443 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.373410    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs"]
Sep 16 21:22:28.378365 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:28.378332    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2908a138_2d0e_4eb8_8b35_742e861fb491.slice/crio-3447d7a165fcb6e9f83aea081db62638bcaf054318cbf0199fe9c6a5a40b79c6 WatchSource:0}: Error finding container 3447d7a165fcb6e9f83aea081db62638bcaf054318cbf0199fe9c6a5a40b79c6: Status 404 returned error can't find the container with id 3447d7a165fcb6e9f83aea081db62638bcaf054318cbf0199fe9c6a5a40b79c6
Sep 16 21:22:28.385702 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.385676    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv"]
Sep 16 21:22:28.389360 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:28.389328    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5258e8e0_274b_4247_966f_99e56319eb18.slice/crio-674f29a9c7c99f92ff8c92edf09036da5fdf60728bcc5a547f39bc972dffbf54 WatchSource:0}: Error finding container 674f29a9c7c99f92ff8c92edf09036da5fdf60728bcc5a547f39bc972dffbf54: Status 404 returned error can't find the container with id 674f29a9c7c99f92ff8c92edf09036da5fdf60728bcc5a547f39bc972dffbf54
Sep 16 21:22:28.406548 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.406504    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq"]
Sep 16 21:22:28.408414 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.408239    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-8p9nq"]
Sep 16 21:22:28.410153 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:28.410126    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2378ae2_b1f3_41b3_a99c_1aa20b008f68.slice/crio-d1eb1fdd208cd8d6edfe7bf12364a54c1f688e2d0df1cb0203a522cc263ffdf5 WatchSource:0}: Error finding container d1eb1fdd208cd8d6edfe7bf12364a54c1f688e2d0df1cb0203a522cc263ffdf5: Status 404 returned error can't find the container with id d1eb1fdd208cd8d6edfe7bf12364a54c1f688e2d0df1cb0203a522cc263ffdf5
Sep 16 21:22:28.410695 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:28.410674    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d6ae3a_7648_4389_9173_9b6487ae023d.slice/crio-b9743e45b8e7236b4b23c267867988d24d7910b80ffe10cccab4598e542828f1 WatchSource:0}: Error finding container b9743e45b8e7236b4b23c267867988d24d7910b80ffe10cccab4598e542828f1: Status 404 returned error can't find the container with id b9743e45b8e7236b4b23c267867988d24d7910b80ffe10cccab4598e542828f1
Sep 16 21:22:28.424037 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.424002    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhhbw"]
Sep 16 21:22:28.427661 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:28.427632    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2c6551_1309_467a_8886_a3c278d89c03.slice/crio-4ac494a0eb853e43a87be4682ad0b5fc95ea77581e4ead1527a53a75760b7d43 WatchSource:0}: Error finding container 4ac494a0eb853e43a87be4682ad0b5fc95ea77581e4ead1527a53a75760b7d43: Status 404 returned error can't find the container with id 4ac494a0eb853e43a87be4682ad0b5fc95ea77581e4ead1527a53a75760b7d43
Sep 16 21:22:28.430772 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.430743    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:28.430865 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.430836    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:28.430865 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.430861    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:28.430997 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.430947    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 21:22:28.430997 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.430980    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 21:22:28.430997 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.430976    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:28.430997 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.430995    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f97687479-dkdrq: secret "image-registry-tls" not found
Sep 16 21:22:28.431183 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.431002    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:29.430978949 +0000 UTC m=+36.079390934 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : secret "router-metrics-certs-default" not found
Sep 16 21:22:28.431183 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.431030    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls podName:9c2b47d0-5a18-4998-824f-e8d425e414c3 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:29.431019389 +0000 UTC m=+36.079431372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls") pod "image-registry-5f97687479-dkdrq" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3") : secret "image-registry-tls" not found
Sep 16 21:22:28.431183 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.431061    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:29.431039629 +0000 UTC m=+36.079451609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : configmap references non-existent config key: service-ca.crt
Sep 16 21:22:28.431183 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.431072    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:28.431183 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.431115    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls podName:2cfc325f-85d2-4a82-aa0e-12c2ff6beb14 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:29.431103919 +0000 UTC m=+36.079515899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5ldzz" (UID: "2cfc325f-85d2-4a82-aa0e-12c2ff6beb14") : secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:28.431183 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.431135    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:28.431183 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:28.431180    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:28.431463 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.431239    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 21:22:28.431463 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.431277    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls podName:f980eff7-4c6c-43da-b2e6-f62f01bda7ee nodeName:}" failed. No retries permitted until 2026-09-16 21:22:29.43126587 +0000 UTC m=+36.079677853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rb794" (UID: "f980eff7-4c6c-43da-b2e6-f62f01bda7ee") : secret "samples-operator-tls" not found
Sep 16 21:22:28.431463 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.431280    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 21:22:28.431463 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:28.431307    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls podName:1e8049ff-a69e-4607-b24f-b98b40f53fc2 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:29.431300634 +0000 UTC m=+36.079712605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls") pod "dns-default-45xsc" (UID: "1e8049ff-a69e-4607-b24f-b98b40f53fc2") : secret "dns-default-metrics-tls" not found
Sep 16 21:22:29.093240 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.093164    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq" event={"ID":"f2378ae2-b1f3-41b3-a99c-1aa20b008f68","Type":"ContainerStarted","Data":"d1eb1fdd208cd8d6edfe7bf12364a54c1f688e2d0df1cb0203a522cc263ffdf5"}
Sep 16 21:22:29.094677 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.094606    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhhbw" event={"ID":"5c2c6551-1309-467a-8886-a3c278d89c03","Type":"ContainerStarted","Data":"4ac494a0eb853e43a87be4682ad0b5fc95ea77581e4ead1527a53a75760b7d43"}
Sep 16 21:22:29.096478 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.096412    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv" event={"ID":"5258e8e0-274b-4247-966f-99e56319eb18","Type":"ContainerStarted","Data":"674f29a9c7c99f92ff8c92edf09036da5fdf60728bcc5a547f39bc972dffbf54"}
Sep 16 21:22:29.098095 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.098064    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn" event={"ID":"9166aeb5-51d3-4a3e-8dbb-c6ff8b46ac1a","Type":"ContainerStarted","Data":"76f07b791447ed7cfbdaf3eb2572fe3687327bd24d6a437d8de8047a298fb06e"}
Sep 16 21:22:29.100072 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.100043    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs" event={"ID":"2908a138-2d0e-4eb8-8b35-742e861fb491","Type":"ContainerStarted","Data":"3447d7a165fcb6e9f83aea081db62638bcaf054318cbf0199fe9c6a5a40b79c6"}
Sep 16 21:22:29.101994 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.101966    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq" event={"ID":"77d6ae3a-7648-4389-9173-9b6487ae023d","Type":"ContainerStarted","Data":"b9743e45b8e7236b4b23c267867988d24d7910b80ffe10cccab4598e542828f1"}
Sep 16 21:22:29.105119 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.104731    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s2szh" event={"ID":"9f518039-df50-4fbb-9d28-f6e554a1a784","Type":"ContainerStarted","Data":"54ce67a636f58077eaa3f4b6389f389643a5acf69d6186847f74317122c8cdda"}
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.440209    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.440305    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.440337    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.440400    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.440435    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:29.440474    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.440626    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.440689    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls podName:1e8049ff-a69e-4607-b24f-b98b40f53fc2 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:31.440669577 +0000 UTC m=+38.089081557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls") pod "dns-default-45xsc" (UID: "1e8049ff-a69e-4607-b24f-b98b40f53fc2") : secret "dns-default-metrics-tls" not found
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441146    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:31.441128702 +0000 UTC m=+38.089540683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : configmap references non-existent config key: service-ca.crt
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441223    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441258    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:31.441247087 +0000 UTC m=+38.089659068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : secret "router-metrics-certs-default" not found
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441321    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441332    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f97687479-dkdrq: secret "image-registry-tls" not found
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441360    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls podName:9c2b47d0-5a18-4998-824f-e8d425e414c3 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:31.441350249 +0000 UTC m=+38.089762228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls") pod "image-registry-5f97687479-dkdrq" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3") : secret "image-registry-tls" not found
Sep 16 21:22:29.441644 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441413    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:29.442872 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441443    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls podName:2cfc325f-85d2-4a82-aa0e-12c2ff6beb14 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:31.441433562 +0000 UTC m=+38.089845543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5ldzz" (UID: "2cfc325f-85d2-4a82-aa0e-12c2ff6beb14") : secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:29.442872 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441494    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 21:22:29.442872 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:29.441523    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls podName:f980eff7-4c6c-43da-b2e6-f62f01bda7ee nodeName:}" failed. No retries permitted until 2026-09-16 21:22:31.441512561 +0000 UTC m=+38.089924542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rb794" (UID: "f980eff7-4c6c-43da-b2e6-f62f01bda7ee") : secret "samples-operator-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:31.465603    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:31.465661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:31.465698    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:31.465747    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:31.465803    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:31.465839    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.465992    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466009    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f97687479-dkdrq: secret "image-registry-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466067    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls podName:9c2b47d0-5a18-4998-824f-e8d425e414c3 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:35.46604906 +0000 UTC m=+42.114461041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls") pod "image-registry-5f97687479-dkdrq" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3") : secret "image-registry-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466544    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466620    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls podName:1e8049ff-a69e-4607-b24f-b98b40f53fc2 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:35.466601643 +0000 UTC m=+42.115013622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls") pod "dns-default-45xsc" (UID: "1e8049ff-a69e-4607-b24f-b98b40f53fc2") : secret "dns-default-metrics-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466680    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466717    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls podName:2cfc325f-85d2-4a82-aa0e-12c2ff6beb14 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:35.466704324 +0000 UTC m=+42.115116304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5ldzz" (UID: "2cfc325f-85d2-4a82-aa0e-12c2ff6beb14") : secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466771    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466802    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls podName:f980eff7-4c6c-43da-b2e6-f62f01bda7ee nodeName:}" failed. No retries permitted until 2026-09-16 21:22:35.466792418 +0000 UTC m=+42.115204398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rb794" (UID: "f980eff7-4c6c-43da-b2e6-f62f01bda7ee") : secret "samples-operator-tls" not found
Sep 16 21:22:31.467056 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466863    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 21:22:31.468530 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466892    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:35.466882062 +0000 UTC m=+42.115294042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : secret "router-metrics-certs-default" not found
Sep 16 21:22:31.468530 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:31.466970    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:35.466960429 +0000 UTC m=+42.115372410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : configmap references non-existent config key: service-ca.crt
Sep 16 21:22:32.866361 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:32.866301    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:32.964125 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:32.964084    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:35.295995 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:35.295885    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:35.368623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:35.368580    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:35.503685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:35.503647    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:35.503685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:35.503699    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:35.503970 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:35.503769    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:35.503970 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.503813    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 21:22:35.503970 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.503860    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 21:22:35.503970 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.503873    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f97687479-dkdrq: secret "image-registry-tls" not found
Sep 16 21:22:35.503970 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.503894    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.503872502 +0000 UTC m=+50.152284488 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : secret "router-metrics-certs-default" not found
Sep 16 21:22:35.503970 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.503919    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:35.503970 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:35.503934    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:35.504299 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.503986    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls podName:9c2b47d0-5a18-4998-824f-e8d425e414c3 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.503951292 +0000 UTC m=+50.152363273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls") pod "image-registry-5f97687479-dkdrq" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3") : secret "image-registry-tls" not found
Sep 16 21:22:35.504299 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.504013    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 21:22:35.504299 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.504020    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls podName:2cfc325f-85d2-4a82-aa0e-12c2ff6beb14 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.504008169 +0000 UTC m=+50.152420142 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5ldzz" (UID: "2cfc325f-85d2-4a82-aa0e-12c2ff6beb14") : secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:35.504299 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.504058    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls podName:f980eff7-4c6c-43da-b2e6-f62f01bda7ee nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.504046241 +0000 UTC m=+50.152458224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rb794" (UID: "f980eff7-4c6c-43da-b2e6-f62f01bda7ee") : secret "samples-operator-tls" not found
Sep 16 21:22:35.504299 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:35.504079    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:35.504299 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:35.504130    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:35.504299 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.504203    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 21:22:35.504299 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.504247    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls podName:1e8049ff-a69e-4607-b24f-b98b40f53fc2 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.504234461 +0000 UTC m=+50.152646442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls") pod "dns-default-45xsc" (UID: "1e8049ff-a69e-4607-b24f-b98b40f53fc2") : secret "dns-default-metrics-tls" not found
Sep 16 21:22:35.504299 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:35.504262    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.504254809 +0000 UTC m=+50.152666776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : configmap references non-existent config key: service-ca.crt
Sep 16 21:22:39.138869 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.138812    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq" event={"ID":"77d6ae3a-7648-4389-9173-9b6487ae023d","Type":"ContainerStarted","Data":"fe1bde375beb4b7de055ecd56081af2d5f766aa45095bda1c18ffea288778287"}
Sep 16 21:22:39.141094 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.140868    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s2szh" event={"ID":"9f518039-df50-4fbb-9d28-f6e554a1a784","Type":"ContainerStarted","Data":"1940bf4f958a179f7f09b3468a67d46e2715ce29c8d5e91cab114f5458f95afc"}
Sep 16 21:22:39.142864 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.142839    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/0.log"
Sep 16 21:22:39.142987 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.142883    2567 generic.go:356] "Generic (PLEG): container finished" podID="f2378ae2-b1f3-41b3-a99c-1aa20b008f68" containerID="48207b2e45f768b29cc1df106a5988e6882c280225cefb6d3a85706a0a2c0488" exitCode=255
Sep 16 21:22:39.142987 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.142977    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq" event={"ID":"f2378ae2-b1f3-41b3-a99c-1aa20b008f68","Type":"ContainerDied","Data":"48207b2e45f768b29cc1df106a5988e6882c280225cefb6d3a85706a0a2c0488"}
Sep 16 21:22:39.143181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.143164    2567 scope.go:117] "RemoveContainer" containerID="48207b2e45f768b29cc1df106a5988e6882c280225cefb6d3a85706a0a2c0488"
Sep 16 21:22:39.146315 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.146288    2567 generic.go:356] "Generic (PLEG): container finished" podID="77e43015-e830-4e21-afed-16ddd27263e9" containerID="7125e1862426e4ea4bcefd165043fe9389dc1b3207ec7db5913dab2670c3cb9c" exitCode=0
Sep 16 21:22:39.146425 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.146376    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgq4m" event={"ID":"77e43015-e830-4e21-afed-16ddd27263e9","Type":"ContainerDied","Data":"7125e1862426e4ea4bcefd165043fe9389dc1b3207ec7db5913dab2670c3cb9c"}
Sep 16 21:22:39.148397 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.148373    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhhbw" event={"ID":"5c2c6551-1309-467a-8886-a3c278d89c03","Type":"ContainerStarted","Data":"bdb096696b44be71397573e163822e3dfd1a4c0d9c025fc48e670c989797bf11"}
Sep 16 21:22:39.150473 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.150449    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv" event={"ID":"5258e8e0-274b-4247-966f-99e56319eb18","Type":"ContainerStarted","Data":"3794a1545686fa37a7b2a2deb04b6bbf98ee4f0067b37cde02041eddf9c33bf2"}
Sep 16 21:22:39.152345 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.152321    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn" event={"ID":"9166aeb5-51d3-4a3e-8dbb-c6ff8b46ac1a","Type":"ContainerStarted","Data":"33e952f6189a979f32466fe0edc9fd131997b32fa5a9887ecd43d2030034831c"}
Sep 16 21:22:39.154283 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.154264    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs" event={"ID":"2908a138-2d0e-4eb8-8b35-742e861fb491","Type":"ContainerStarted","Data":"8bd26a8f2491a1730c0bf173afdf9cce165124f0522a3556e3dde12d146e9457"}
Sep 16 21:22:39.155843 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.155799    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-c4ckq" podStartSLOduration=7.273483314 podStartE2EDuration="17.155784597s" totalImagesPullingTime="9.882301283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:39.154880784 +0000 UTC m=+45.803292774" watchObservedRunningTime="2026-09-16 21:22:39.155784597 +0000 UTC m=+45.804196582"
Sep 16 21:22:39.223865 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.223196    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-s2szh" podStartSLOduration=7.262651123 podStartE2EDuration="17.223175897s" totalImagesPullingTime="9.960524774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:39.198502268 +0000 UTC m=+45.846914257" watchObservedRunningTime="2026-09-16 21:22:39.223175897 +0000 UTC m=+45.871587886"
Sep 16 21:22:39.223865 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.223342    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jhhbw" podStartSLOduration=2.368978725 podStartE2EDuration="12.22333221s" totalImagesPullingTime="9.854353485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:39.22332247 +0000 UTC m=+45.871734463" watchObservedRunningTime="2026-09-16 21:22:39.22333221 +0000 UTC m=+45.871744199"
Sep 16 21:22:39.273943 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.272808    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-skqzn" podStartSLOduration=7.822423362 podStartE2EDuration="17.272787101s" totalImagesPullingTime="9.450363739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:39.270121768 +0000 UTC m=+45.918533756" watchObservedRunningTime="2026-09-16 21:22:39.272787101 +0000 UTC m=+45.921199091"
Sep 16 21:22:39.295200 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.287440    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv" podStartSLOduration=7.394881283 podStartE2EDuration="17.287418296s" totalImagesPullingTime="9.892537013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:39.287081819 +0000 UTC m=+45.935493800" watchObservedRunningTime="2026-09-16 21:22:39.287418296 +0000 UTC m=+45.935830278"
Sep 16 21:22:39.306801 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.306115    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs" podStartSLOduration=7.403472034 podStartE2EDuration="17.306089597s" totalImagesPullingTime="9.902617563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:39.304611614 +0000 UTC m=+45.953023603" watchObservedRunningTime="2026-09-16 21:22:39.306089597 +0000 UTC m=+45.954501586"
Sep 16 21:22:39.351274 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.351200    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:39.451507 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.451470    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:39.706164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.706082    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tdnb5"]
Sep 16 21:22:39.727680 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.727646    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdnb5"]
Sep 16 21:22:39.727875 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.727833    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.729476 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.729452    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:39.730700 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.730614    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 21:22:39.730700 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.730654    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 21:22:39.730700 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.730681    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hr64t\""
Sep 16 21:22:39.746261 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.746235    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swgxt\" (UniqueName: \"kubernetes.io/projected/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-api-access-swgxt\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.746395 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.746329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.746395 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.746381    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.746506 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.746410    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-data-volume\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.746594 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.746571    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-crio-socket\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.791710 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.791666    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:39.847964 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.847923    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.848179 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.848019    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.848179 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.848059    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-data-volume\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.848179 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:39.848073    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:39.848179 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.848128    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-crio-socket\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.848179 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:39.848145    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls podName:9093c52e-a8f7-4d7b-858f-34d04fea5ebd nodeName:}" failed. No retries permitted until 2026-09-16 21:22:40.348128861 +0000 UTC m=+46.996540827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdnb5" (UID: "9093c52e-a8f7-4d7b-858f-34d04fea5ebd") : secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:39.848179 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:39.848153    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 21:22:39.848430 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.848205    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swgxt\" (UniqueName: \"kubernetes.io/projected/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-api-access-swgxt\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.848430 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:39.848214    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm podName:9093c52e-a8f7-4d7b-858f-34d04fea5ebd nodeName:}" failed. No retries permitted until 2026-09-16 21:22:40.348196953 +0000 UTC m=+46.996608930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdnb5" (UID: "9093c52e-a8f7-4d7b-858f-34d04fea5ebd") : configmap "kube-rbac-proxy" not found
Sep 16 21:22:39.848430 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.848319    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-crio-socket\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.848532 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.848469    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-data-volume\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:39.857396 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:39.857366    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swgxt\" (UniqueName: \"kubernetes.io/projected/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-api-access-swgxt\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:40.069678 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.069642    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:40.132272 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.132220    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:40.159533 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.159505    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:22:40.159973 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.159957    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/0.log"
Sep 16 21:22:40.160043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.159994    2567 generic.go:356] "Generic (PLEG): container finished" podID="f2378ae2-b1f3-41b3-a99c-1aa20b008f68" containerID="a4f8933df5ad839db0cb784884199ff7e0ba6b42b2118ef0dd6c1a76f085ee0e" exitCode=255
Sep 16 21:22:40.160096 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.160066    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq" event={"ID":"f2378ae2-b1f3-41b3-a99c-1aa20b008f68","Type":"ContainerDied","Data":"a4f8933df5ad839db0cb784884199ff7e0ba6b42b2118ef0dd6c1a76f085ee0e"}
Sep 16 21:22:40.160140 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.160117    2567 scope.go:117] "RemoveContainer" containerID="48207b2e45f768b29cc1df106a5988e6882c280225cefb6d3a85706a0a2c0488"
Sep 16 21:22:40.160372 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.160349    2567 scope.go:117] "RemoveContainer" containerID="a4f8933df5ad839db0cb784884199ff7e0ba6b42b2118ef0dd6c1a76f085ee0e"
Sep 16 21:22:40.160573 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:40.160554    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8p9nq_openshift-console-operator(f2378ae2-b1f3-41b3-a99c-1aa20b008f68)\"" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq" podUID="f2378ae2-b1f3-41b3-a99c-1aa20b008f68"
Sep 16 21:22:40.163620 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.163587    2567 generic.go:356] "Generic (PLEG): container finished" podID="77e43015-e830-4e21-afed-16ddd27263e9" containerID="951ce953201bbcd782e93a14a1a6ad9e218a226eb8cdbe34cacb65656596b0ae" exitCode=0
Sep 16 21:22:40.163714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.163681    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgq4m" event={"ID":"77e43015-e830-4e21-afed-16ddd27263e9","Type":"ContainerDied","Data":"951ce953201bbcd782e93a14a1a6ad9e218a226eb8cdbe34cacb65656596b0ae"}
Sep 16 21:22:40.207308 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.207276    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:40.285266 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.285231    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:40.320816 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.320741    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz"]
Sep 16 21:22:40.339643 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.339603    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz"]
Sep 16 21:22:40.339803 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.339728    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz"
Sep 16 21:22:40.342159 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.342137    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 21:22:40.342305 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.342288    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 21:22:40.342345 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.342292    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-dd8bl\""
Sep 16 21:22:40.355716 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.355689    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:40.355848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.355741    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:40.355848 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:40.355840    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:40.355932 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:40.355855    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 21:22:40.355932 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:40.355927    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls podName:9093c52e-a8f7-4d7b-858f-34d04fea5ebd nodeName:}" failed. No retries permitted until 2026-09-16 21:22:41.355889481 +0000 UTC m=+48.004301447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdnb5" (UID: "9093c52e-a8f7-4d7b-858f-34d04fea5ebd") : secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:40.356009 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:40.355947    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm podName:9093c52e-a8f7-4d7b-858f-34d04fea5ebd nodeName:}" failed. No retries permitted until 2026-09-16 21:22:41.355939215 +0000 UTC m=+48.004351195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdnb5" (UID: "9093c52e-a8f7-4d7b-858f-34d04fea5ebd") : configmap "kube-rbac-proxy" not found
Sep 16 21:22:40.365399 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.365373    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:40.424144 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.424110    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:40.456441 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.456401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54bgl\" (UniqueName: \"kubernetes.io/projected/a64f28e2-ac14-4d5a-a440-f602bca5222e-kube-api-access-54bgl\") pod \"migrator-8b4d697c5-vv6hz\" (UID: \"a64f28e2-ac14-4d5a-a440-f602bca5222e\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz"
Sep 16 21:22:40.557635 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.557600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54bgl\" (UniqueName: \"kubernetes.io/projected/a64f28e2-ac14-4d5a-a440-f602bca5222e-kube-api-access-54bgl\") pod \"migrator-8b4d697c5-vv6hz\" (UID: \"a64f28e2-ac14-4d5a-a440-f602bca5222e\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz"
Sep 16 21:22:40.567594 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.567571    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54bgl\" (UniqueName: \"kubernetes.io/projected/a64f28e2-ac14-4d5a-a440-f602bca5222e-kube-api-access-54bgl\") pod \"migrator-8b4d697c5-vv6hz\" (UID: \"a64f28e2-ac14-4d5a-a440-f602bca5222e\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz"
Sep 16 21:22:40.648697 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.648626    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz"
Sep 16 21:22:40.768305 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.768266    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:40.868499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.868457    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:40.884319 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.884293    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xd599_5fb0449b-4fbe-446e-8aa0-9d2c653d440d/dns-node-resolver/0.log"
Sep 16 21:22:40.934303 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:40.934281    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz"]
Sep 16 21:22:40.936635 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:40.936606    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64f28e2_ac14_4d5a_a440_f602bca5222e.slice/crio-1c84371c275a39cfcab907c2e3877432e2ef89b6cf7aa475aba1f86a2d40438f WatchSource:0}: Error finding container 1c84371c275a39cfcab907c2e3877432e2ef89b6cf7aa475aba1f86a2d40438f: Status 404 returned error can't find the container with id 1c84371c275a39cfcab907c2e3877432e2ef89b6cf7aa475aba1f86a2d40438f
Sep 16 21:22:41.169070 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.169028    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz" event={"ID":"a64f28e2-ac14-4d5a-a440-f602bca5222e","Type":"ContainerStarted","Data":"1c84371c275a39cfcab907c2e3877432e2ef89b6cf7aa475aba1f86a2d40438f"}
Sep 16 21:22:41.170609 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.170587    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:22:41.170971 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.170949    2567 scope.go:117] "RemoveContainer" containerID="a4f8933df5ad839db0cb784884199ff7e0ba6b42b2118ef0dd6c1a76f085ee0e"
Sep 16 21:22:41.171132 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:41.171109    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8p9nq_openshift-console-operator(f2378ae2-b1f3-41b3-a99c-1aa20b008f68)\"" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq" podUID="f2378ae2-b1f3-41b3-a99c-1aa20b008f68"
Sep 16 21:22:41.173968 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.173945    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hgq4m" event={"ID":"77e43015-e830-4e21-afed-16ddd27263e9","Type":"ContainerStarted","Data":"e57e7524dd2f69562c0e599c8d39de8da0d108654def698d1b52b5840ace24fe"}
Sep 16 21:22:41.221787 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.221691    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hgq4m" podStartSLOduration=9.65977766 podStartE2EDuration="47.221675781s" totalImagesPullingTime="37.561898121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:41.219743263 +0000 UTC m=+47.868155252" watchObservedRunningTime="2026-09-16 21:22:41.221675781 +0000 UTC m=+47.870087762"
Sep 16 21:22:41.341365 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.341329    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:41.364598 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.364563    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:41.364745 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.364628    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:41.364745 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:41.364706    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:41.364841 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:41.364767    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls podName:9093c52e-a8f7-4d7b-858f-34d04fea5ebd nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.364751319 +0000 UTC m=+50.013163297 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdnb5" (UID: "9093c52e-a8f7-4d7b-858f-34d04fea5ebd") : secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:41.364841 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:41.364703    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 21:22:41.364841 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:41.364801    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm podName:9093c52e-a8f7-4d7b-858f-34d04fea5ebd nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.364794882 +0000 UTC m=+50.013206885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdnb5" (UID: "9093c52e-a8f7-4d7b-858f-34d04fea5ebd") : configmap "kube-rbac-proxy" not found
Sep 16 21:22:41.441656 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.441625    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:41.491868 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.491792    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kfcgt_b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5/node-ca/0.log"
Sep 16 21:22:41.693235 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.693196    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:41.767448 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.767409    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:41.846054 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.846021    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:41.923557 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.923524    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:41.927071 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.927044    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"]
Sep 16 21:22:41.953495 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.953459    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"]
Sep 16 21:22:41.953679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.953586    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:41.956322 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.956297    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 16 21:22:41.956471 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.956340    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 16 21:22:41.956471 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:41.956379    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bprzq\""
Sep 16 21:22:42.007438 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.007399    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:42.070565 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.070478    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:42.070711 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.070658    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f40f29c-8841-4a0b-8b1a-343e91436f77-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:42.083277 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.083248    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhhbw_5c2c6551-1309-467a-8886-a3c278d89c03/serve-healthcheck-canary/0.log"
Sep 16 21:22:42.171879 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.171845    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f40f29c-8841-4a0b-8b1a-343e91436f77-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:42.172376 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.171971    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:42.172376 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:42.172248    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 21:22:42.172376 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:42.172312    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert podName:8f40f29c-8841-4a0b-8b1a-343e91436f77 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:42.67229325 +0000 UTC m=+49.320705217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ndwmx" (UID: "8f40f29c-8841-4a0b-8b1a-343e91436f77") : secret "networking-console-plugin-cert" not found
Sep 16 21:22:42.172619 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.172582    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8f40f29c-8841-4a0b-8b1a-343e91436f77-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:42.312940 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.312897    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:42.361151 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.361072    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-fpzlz"]
Sep 16 21:22:42.373971 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.373949    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-fpzlz"]
Sep 16 21:22:42.374096 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.374008    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.376497 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.376472    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-wzwst\""
Sep 16 21:22:42.376627 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.376476    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 21:22:42.376627 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.376513    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 21:22:42.377421 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.377404    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 21:22:42.377510 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.377453    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 21:22:42.392776 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.392750    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:42.457232 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.457194    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:42.474353 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.474320    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/470a6a77-601f-46f7-a76d-47c324a2626a-signing-key\") pod \"service-ca-568869fdc5-fpzlz\" (UID: \"470a6a77-601f-46f7-a76d-47c324a2626a\") " pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.474558 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.474533    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjdxg\" (UniqueName: \"kubernetes.io/projected/470a6a77-601f-46f7-a76d-47c324a2626a-kube-api-access-hjdxg\") pod \"service-ca-568869fdc5-fpzlz\" (UID: \"470a6a77-601f-46f7-a76d-47c324a2626a\") " pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.474678 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.474652    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/470a6a77-601f-46f7-a76d-47c324a2626a-signing-cabundle\") pod \"service-ca-568869fdc5-fpzlz\" (UID: \"470a6a77-601f-46f7-a76d-47c324a2626a\") " pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.575363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.575316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjdxg\" (UniqueName: \"kubernetes.io/projected/470a6a77-601f-46f7-a76d-47c324a2626a-kube-api-access-hjdxg\") pod \"service-ca-568869fdc5-fpzlz\" (UID: \"470a6a77-601f-46f7-a76d-47c324a2626a\") " pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.575585 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.575434    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/470a6a77-601f-46f7-a76d-47c324a2626a-signing-cabundle\") pod \"service-ca-568869fdc5-fpzlz\" (UID: \"470a6a77-601f-46f7-a76d-47c324a2626a\") " pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.575585 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.575518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/470a6a77-601f-46f7-a76d-47c324a2626a-signing-key\") pod \"service-ca-568869fdc5-fpzlz\" (UID: \"470a6a77-601f-46f7-a76d-47c324a2626a\") " pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.576315 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.576287    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/470a6a77-601f-46f7-a76d-47c324a2626a-signing-cabundle\") pod \"service-ca-568869fdc5-fpzlz\" (UID: \"470a6a77-601f-46f7-a76d-47c324a2626a\") " pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.578034 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.578007    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/470a6a77-601f-46f7-a76d-47c324a2626a-signing-key\") pod \"service-ca-568869fdc5-fpzlz\" (UID: \"470a6a77-601f-46f7-a76d-47c324a2626a\") " pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.583705 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.583686    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjdxg\" (UniqueName: \"kubernetes.io/projected/470a6a77-601f-46f7-a76d-47c324a2626a-kube-api-access-hjdxg\") pod \"service-ca-568869fdc5-fpzlz\" (UID: \"470a6a77-601f-46f7-a76d-47c324a2626a\") " pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.676127 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.676045    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:42.676273 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:42.676190    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 21:22:42.676273 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:42.676262    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert podName:8f40f29c-8841-4a0b-8b1a-343e91436f77 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:43.67624493 +0000 UTC m=+50.324656896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ndwmx" (UID: "8f40f29c-8841-4a0b-8b1a-343e91436f77") : secret "networking-console-plugin-cert" not found
Sep 16 21:22:42.682943 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.682921    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-fpzlz"
Sep 16 21:22:42.685624 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.685596    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:42.776835 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.776373    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:42.996815 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:42.996740    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-fpzlz"]
Sep 16 21:22:43.107805 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:43.107776    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470a6a77_601f_46f7_a76d_47c324a2626a.slice/crio-f79df97b69ba3dea2e0f0d968b86e7f11acb4f488152e8fcb50480a81c2a9804 WatchSource:0}: Error finding container f79df97b69ba3dea2e0f0d968b86e7f11acb4f488152e8fcb50480a81c2a9804: Status 404 returned error can't find the container with id f79df97b69ba3dea2e0f0d968b86e7f11acb4f488152e8fcb50480a81c2a9804
Sep 16 21:22:43.182254 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.182225    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-fpzlz" event={"ID":"470a6a77-601f-46f7-a76d-47c324a2626a","Type":"ContainerStarted","Data":"f79df97b69ba3dea2e0f0d968b86e7f11acb4f488152e8fcb50480a81c2a9804"}
Sep 16 21:22:43.383323 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.383291    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:43.383476 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.383352    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:43.383560 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.383486    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:43.383628 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.383573    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls podName:9093c52e-a8f7-4d7b-858f-34d04fea5ebd nodeName:}" failed. No retries permitted until 2026-09-16 21:22:47.383554786 +0000 UTC m=+54.031966768 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdnb5" (UID: "9093c52e-a8f7-4d7b-858f-34d04fea5ebd") : secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:43.383628 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.383489    2567 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 21:22:43.383742 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.383633    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm podName:9093c52e-a8f7-4d7b-858f-34d04fea5ebd nodeName:}" failed. No retries permitted until 2026-09-16 21:22:47.383616277 +0000 UTC m=+54.032028260 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tdnb5" (UID: "9093c52e-a8f7-4d7b-858f-34d04fea5ebd") : configmap "kube-rbac-proxy" not found
Sep 16 21:22:43.492219 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.492182    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:43.574290 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.574201    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:43.584416 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.584385    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:43.584570 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.584462    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:43.584570 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.584490    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:43.584570 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584555    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:59.584536017 +0000 UTC m=+66.232947999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : configmap references non-existent config key: service-ca.crt
Sep 16 21:22:43.584709 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.584578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:43.584709 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584612    2567 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 21:22:43.584709 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584613    2567 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 16 21:22:43.584709 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.584626    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:43.584709 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584675    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs podName:863ab98a-8183-4e24-aba7-8d57e12368c6 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:59.584656968 +0000 UTC m=+66.233068948 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs") pod "router-default-5f57db6868-lgg22" (UID: "863ab98a-8183-4e24-aba7-8d57e12368c6") : secret "router-metrics-certs-default" not found
Sep 16 21:22:43.584709 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584678    2567 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:43.584709 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584625    2567 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f97687479-dkdrq: secret "image-registry-tls" not found
Sep 16 21:22:43.584709 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584709    2567 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 16 21:22:43.585061 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584716    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls podName:2cfc325f-85d2-4a82-aa0e-12c2ff6beb14 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:59.584704658 +0000 UTC m=+66.233116647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-5ldzz" (UID: "2cfc325f-85d2-4a82-aa0e-12c2ff6beb14") : secret "cluster-monitoring-operator-tls" not found
Sep 16 21:22:43.585061 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584744    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls podName:f980eff7-4c6c-43da-b2e6-f62f01bda7ee nodeName:}" failed. No retries permitted until 2026-09-16 21:22:59.584734336 +0000 UTC m=+66.233146319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-rb794" (UID: "f980eff7-4c6c-43da-b2e6-f62f01bda7ee") : secret "samples-operator-tls" not found
Sep 16 21:22:43.585061 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584754    2567 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 21:22:43.585061 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.584700    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:43.585061 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584791    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls podName:9c2b47d0-5a18-4998-824f-e8d425e414c3 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:59.584769991 +0000 UTC m=+66.233181970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls") pod "image-registry-5f97687479-dkdrq" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3") : secret "image-registry-tls" not found
Sep 16 21:22:43.585061 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.584818    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls podName:1e8049ff-a69e-4607-b24f-b98b40f53fc2 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:59.58480785 +0000 UTC m=+66.233219821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls") pod "dns-default-45xsc" (UID: "1e8049ff-a69e-4607-b24f-b98b40f53fc2") : secret "dns-default-metrics-tls" not found
Sep 16 21:22:43.669151 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.669114    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:43.685448 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.685421    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:43.685561 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.685549    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 21:22:43.685636 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:43.685617    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert podName:8f40f29c-8841-4a0b-8b1a-343e91436f77 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:45.685598615 +0000 UTC m=+52.334010597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ndwmx" (UID: "8f40f29c-8841-4a0b-8b1a-343e91436f77") : secret "networking-console-plugin-cert" not found
Sep 16 21:22:43.898078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.897989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:43.998231 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:43.998200    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:44.188469 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.188311    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz" event={"ID":"a64f28e2-ac14-4d5a-a440-f602bca5222e","Type":"ContainerStarted","Data":"cad72fb26703de894797dbacb8f5db83d9c674114363ab354c619237791022cd"}
Sep 16 21:22:44.188469 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.188391    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz" event={"ID":"a64f28e2-ac14-4d5a-a440-f602bca5222e","Type":"ContainerStarted","Data":"4ed4047cdba43b685620dfafdb0fcdfc2c9c22a8a4d9d767a71b823e4cda798d"}
Sep 16 21:22:44.190246 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.190220    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-fpzlz" event={"ID":"470a6a77-601f-46f7-a76d-47c324a2626a","Type":"ContainerStarted","Data":"3dc1ab2eed98093e8f828532c2febf2bc71a28c651979f86bfa67d99602236f3"}
Sep 16 21:22:44.205531 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.205475    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-vv6hz" podStartSLOduration=2.011651719 podStartE2EDuration="4.205459818s" totalImagesPullingTime="2.193808099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:44.204934956 +0000 UTC m=+50.853346945" watchObservedRunningTime="2026-09-16 21:22:44.205459818 +0000 UTC m=+50.853871838"
Sep 16 21:22:44.221469 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.221420    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-fpzlz" podStartSLOduration=2.221394585 podStartE2EDuration="2.221394585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:44.22057708 +0000 UTC m=+50.868989104" watchObservedRunningTime="2026-09-16 21:22:44.221394585 +0000 UTC m=+50.869806574"
Sep 16 21:22:44.295702 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.295667    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:44.548272 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.548237    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:44.649031 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.648983    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:44.763413 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.763378    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:44.822834 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.822746    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:44.987225 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:44.987191    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:45.704987 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:45.704947    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:45.705386 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:45.705111    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 21:22:45.705386 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:45.705181    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert podName:8f40f29c-8841-4a0b-8b1a-343e91436f77 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:49.705164491 +0000 UTC m=+56.353576457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ndwmx" (UID: "8f40f29c-8841-4a0b-8b1a-343e91436f77") : secret "networking-console-plugin-cert" not found
Sep 16 21:22:45.796050 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:45.796016    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:46.071310 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:46.071278    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:46.191026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:46.190996    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:46.594020 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:46.593974    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:46.797811 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:46.797767    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:47.203305 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:47.203271    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:47.419162 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:47.419129    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:47.419395 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:47.419181    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:47.419395 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:47.419289    2567 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:47.419395 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:47.419358    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls podName:9093c52e-a8f7-4d7b-858f-34d04fea5ebd nodeName:}" failed. No retries permitted until 2026-09-16 21:22:55.419340949 +0000 UTC m=+62.067752936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tdnb5" (UID: "9093c52e-a8f7-4d7b-858f-34d04fea5ebd") : secret "insights-runtime-extractor-tls" not found
Sep 16 21:22:47.419711 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:47.419693    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:47.453723 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:47.453655    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:47.512896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:47.512870    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:47.786665 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:47.786622    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:47.991473 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:47.991437    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:48.070240 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:48.070145    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:48.070240 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:48.070211    2567 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:22:48.070679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:48.070665    2567 scope.go:117] "RemoveContainer" containerID="a4f8933df5ad839db0cb784884199ff7e0ba6b42b2118ef0dd6c1a76f085ee0e"
Sep 16 21:22:48.070879 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:48.070856    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-8p9nq_openshift-console-operator(f2378ae2-b1f3-41b3-a99c-1aa20b008f68)\"" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq" podUID="f2378ae2-b1f3-41b3-a99c-1aa20b008f68"
Sep 16 21:22:48.091984 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:48.091942    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:48.318217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:48.318181    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:48.597374 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:48.597340    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:49.013513 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:49.013479    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:49.074225 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:49.074198    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:49.451606 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:49.451519    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:49.739611 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:49.739524    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:49.739772 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:49.739684    2567 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 16 21:22:49.739772 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:22:49.739753    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert podName:8f40f29c-8841-4a0b-8b1a-343e91436f77 nodeName:}" failed. No retries permitted until 2026-09-16 21:22:57.73973698 +0000 UTC m=+64.388148963 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-ndwmx" (UID: "8f40f29c-8841-4a0b-8b1a-343e91436f77") : secret "networking-console-plugin-cert" not found
Sep 16 21:22:49.849719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:49.849684    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:50.189544 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:50.189507    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:50.293287 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:50.293249    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:50.396208 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:50.396176    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:50.593235 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:50.593202    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:50.693254 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:50.693214    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:50.794292 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:50.794259    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:50.856310 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:50.856230    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:51.400027 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:51.399987    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:51.786318 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:51.786285    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:51.992626 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:51.992584    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:52.391325 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:52.391288    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:52.989183 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:52.989150    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:53.080753 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:53.080730    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-579wp"
Sep 16 21:22:53.197290 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:53.197255    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:53.802678 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:53.802647    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:54.225962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:54.225845    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:54.591118 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:54.591087    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:54.791731 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:54.791689    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:54.993651 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:54.993574    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:22:55.490107 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:55.490072    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:55.492356 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:55.492334    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9093c52e-a8f7-4d7b-858f-34d04fea5ebd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tdnb5\" (UID: \"9093c52e-a8f7-4d7b-858f-34d04fea5ebd\") " pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:55.641458 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:55.641419    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hr64t\""
Sep 16 21:22:55.649527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:55.649501    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tdnb5"
Sep 16 21:22:55.928137 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:55.928046    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tdnb5"]
Sep 16 21:22:55.930624 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:55.930599    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9093c52e_a8f7_4d7b_858f_34d04fea5ebd.slice/crio-968c5b01f9492a70afb027334e96880fea6166d88f9942f2ae1c722db82f6dd0 WatchSource:0}: Error finding container 968c5b01f9492a70afb027334e96880fea6166d88f9942f2ae1c722db82f6dd0: Status 404 returned error can't find the container with id 968c5b01f9492a70afb027334e96880fea6166d88f9942f2ae1c722db82f6dd0
Sep 16 21:22:56.231583 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:56.231503    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdnb5" event={"ID":"9093c52e-a8f7-4d7b-858f-34d04fea5ebd","Type":"ContainerStarted","Data":"fe7485b24103d8888274673239e60164b5a50608784435268072371a0f957343"}
Sep 16 21:22:56.231583 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:56.231556    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdnb5" event={"ID":"9093c52e-a8f7-4d7b-858f-34d04fea5ebd","Type":"ContainerStarted","Data":"968c5b01f9492a70afb027334e96880fea6166d88f9942f2ae1c722db82f6dd0"}
Sep 16 21:22:57.236627 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:57.236589    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdnb5" event={"ID":"9093c52e-a8f7-4d7b-858f-34d04fea5ebd","Type":"ContainerStarted","Data":"7db3a3a8d92a8e41ff26d98a638fb1ae294a149bcd63d4a63e6c47fcd07c4e25"}
Sep 16 21:22:57.812855 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:57.812817    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:57.815235 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:57.815212    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8f40f29c-8841-4a0b-8b1a-343e91436f77-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-ndwmx\" (UID: \"8f40f29c-8841-4a0b-8b1a-343e91436f77\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:57.865398 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:57.865358    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-bprzq\""
Sep 16 21:22:57.873531 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:57.873500    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"
Sep 16 21:22:58.448642 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:58.448520    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx"]
Sep 16 21:22:58.451106 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:22:58.451069    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f40f29c_8841_4a0b_8b1a_343e91436f77.slice/crio-1521219f932d571e1d0b24de09b5b44ec40fe2a26b8660d6c66e56a4289b7828 WatchSource:0}: Error finding container 1521219f932d571e1d0b24de09b5b44ec40fe2a26b8660d6c66e56a4289b7828: Status 404 returned error can't find the container with id 1521219f932d571e1d0b24de09b5b44ec40fe2a26b8660d6c66e56a4289b7828
Sep 16 21:22:59.245563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.245526    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx" event={"ID":"8f40f29c-8841-4a0b-8b1a-343e91436f77","Type":"ContainerStarted","Data":"1521219f932d571e1d0b24de09b5b44ec40fe2a26b8660d6c66e56a4289b7828"}
Sep 16 21:22:59.247812 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.247781    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tdnb5" event={"ID":"9093c52e-a8f7-4d7b-858f-34d04fea5ebd","Type":"ContainerStarted","Data":"094fc34046d50b4e48614a916a82aa91c927c5aa0f61724d33a2ffb2ad1eee65"}
Sep 16 21:22:59.265964 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.265912    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tdnb5" podStartSLOduration=18.090044348 podStartE2EDuration="20.265876836s" totalImagesPullingTime="2.175832488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:22:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:22:59.264053953 +0000 UTC m=+65.912465955" watchObservedRunningTime="2026-09-16 21:22:59.265876836 +0000 UTC m=+65.914288824"
Sep 16 21:22:59.531494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.531455    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:59.534130 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.534104    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 21:22:59.544257 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.544225    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38c2151a-8d75-4772-a0a2-922fb7ca7e20-original-pull-secret\") pod \"global-pull-secret-syncer-dh4v8\" (UID: \"38c2151a-8d75-4772-a0a2-922fb7ca7e20\") " pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:59.632236 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.632211    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:59.632363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.632258    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:59.632363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.632284    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:59.632363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.632308    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:59.632363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.632327    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:59.632363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.632344    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:59.632618 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.632377    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:59.632618 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.632408    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:59.633086 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.633059    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/863ab98a-8183-4e24-aba7-8d57e12368c6-service-ca-bundle\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:59.634733 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.634703    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e8049ff-a69e-4607-b24f-b98b40f53fc2-metrics-tls\") pod \"dns-default-45xsc\" (UID: \"1e8049ff-a69e-4607-b24f-b98b40f53fc2\") " pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:59.634842 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.634811    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 21:22:59.634918 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.634854    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"image-registry-5f97687479-dkdrq\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") " pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:59.635194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.635171    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863ab98a-8183-4e24-aba7-8d57e12368c6-metrics-certs\") pod \"router-default-5f57db6868-lgg22\" (UID: \"863ab98a-8183-4e24-aba7-8d57e12368c6\") " pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:59.635355 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.635334    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2cfc325f-85d2-4a82-aa0e-12c2ff6beb14-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-5ldzz\" (UID: \"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:59.635399 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.635345    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhswz\" (UniqueName: \"kubernetes.io/projected/d4a729ab-f653-4c7c-a0bb-23569feae959-kube-api-access-zhswz\") pod \"network-check-target-frkfh\" (UID: \"d4a729ab-f653-4c7c-a0bb-23569feae959\") " pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:59.635433 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.635407    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f980eff7-4c6c-43da-b2e6-f62f01bda7ee-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-rb794\" (UID: \"f980eff7-4c6c-43da-b2e6-f62f01bda7ee\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:59.645374 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.645352    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffed9181-7dab-40e2-aa2a-94f3ef7dae5d-metrics-certs\") pod \"network-metrics-daemon-x98p4\" (UID: \"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d\") " pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:59.778875 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.778840    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hdj86\""
Sep 16 21:22:59.783783 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.783738    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dh4v8"
Sep 16 21:22:59.786673 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.786656    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:22:59.792576 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.792552    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8xpmt\""
Sep 16 21:22:59.794935 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.794916    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5496m\""
Sep 16 21:22:59.801047 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.801025    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x98p4"
Sep 16 21:22:59.802040 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.802022    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ckjjz\""
Sep 16 21:22:59.803075 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.803052    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:22:59.810120 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.810094    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:22:59.814381 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.814349    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wmdmb\""
Sep 16 21:22:59.819626 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.819555    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"
Sep 16 21:22:59.835970 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.835711    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rcnph\""
Sep 16 21:22:59.839927 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.839884    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7zskn\""
Sep 16 21:22:59.844395 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.844128    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45xsc"
Sep 16 21:22:59.849313 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.849231    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"
Sep 16 21:22:59.936737 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:22:59.936707    2567 scope.go:117] "RemoveContainer" containerID="a4f8933df5ad839db0cb784884199ff7e0ba6b42b2118ef0dd6c1a76f085ee0e"
Sep 16 21:23:00.077373 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.077293    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dh4v8"]
Sep 16 21:23:00.079749 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.079726    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-frkfh"]
Sep 16 21:23:00.080510 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:00.080490    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c2151a_8d75_4772_a0a2_922fb7ca7e20.slice/crio-34b213348675e12b69a3122fac609ccb8eee804d9e357081389db0cd794c9c03 WatchSource:0}: Error finding container 34b213348675e12b69a3122fac609ccb8eee804d9e357081389db0cd794c9c03: Status 404 returned error can't find the container with id 34b213348675e12b69a3122fac609ccb8eee804d9e357081389db0cd794c9c03
Sep 16 21:23:00.083343 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:00.083321    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4a729ab_f653_4c7c_a0bb_23569feae959.slice/crio-982efebf802b2b2d03bb4f72a1f76bcd7529f59fdff33638be31b5cfc979c439 WatchSource:0}: Error finding container 982efebf802b2b2d03bb4f72a1f76bcd7529f59fdff33638be31b5cfc979c439: Status 404 returned error can't find the container with id 982efebf802b2b2d03bb4f72a1f76bcd7529f59fdff33638be31b5cfc979c439
Sep 16 21:23:00.132263 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.132238    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x98p4"]
Sep 16 21:23:00.136469 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.136106    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz"]
Sep 16 21:23:00.138563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.137582    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5f57db6868-lgg22"]
Sep 16 21:23:00.139268 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.139106    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f97687479-dkdrq"]
Sep 16 21:23:00.140119 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:00.140099    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfc325f_85d2_4a82_aa0e_12c2ff6beb14.slice/crio-b733b5dfc7436989e39f696f117a9249bf39d15fedd149aa832e416626341feb WatchSource:0}: Error finding container b733b5dfc7436989e39f696f117a9249bf39d15fedd149aa832e416626341feb: Status 404 returned error can't find the container with id b733b5dfc7436989e39f696f117a9249bf39d15fedd149aa832e416626341feb
Sep 16 21:23:00.141201 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:00.141153    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863ab98a_8183_4e24_aba7_8d57e12368c6.slice/crio-46aa0a669c079a5eb179073b010ca0deb2f3db500b2214bc69a14ec866961315 WatchSource:0}: Error finding container 46aa0a669c079a5eb179073b010ca0deb2f3db500b2214bc69a14ec866961315: Status 404 returned error can't find the container with id 46aa0a669c079a5eb179073b010ca0deb2f3db500b2214bc69a14ec866961315
Sep 16 21:23:00.142833 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:00.142309    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2b47d0_5a18_4998_824f_e8d425e414c3.slice/crio-7c887945c391cd069c21451dd5222f1e0214ac4ace3326b5a10ff9b40cd6c021 WatchSource:0}: Error finding container 7c887945c391cd069c21451dd5222f1e0214ac4ace3326b5a10ff9b40cd6c021: Status 404 returned error can't find the container with id 7c887945c391cd069c21451dd5222f1e0214ac4ace3326b5a10ff9b40cd6c021
Sep 16 21:23:00.196513 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.196487    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45xsc"]
Sep 16 21:23:00.198175 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.197779    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794"]
Sep 16 21:23:00.202225 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:00.202193    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8049ff_a69e_4607_b24f_b98b40f53fc2.slice/crio-71664fc873adb0f9c19b43742a4f1717b4d82d92cd61becb37d56e6335a9edac WatchSource:0}: Error finding container 71664fc873adb0f9c19b43742a4f1717b4d82d92cd61becb37d56e6335a9edac: Status 404 returned error can't find the container with id 71664fc873adb0f9c19b43742a4f1717b4d82d92cd61becb37d56e6335a9edac
Sep 16 21:23:00.253337 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.253296    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45xsc" event={"ID":"1e8049ff-a69e-4607-b24f-b98b40f53fc2","Type":"ContainerStarted","Data":"71664fc873adb0f9c19b43742a4f1717b4d82d92cd61becb37d56e6335a9edac"}
Sep 16 21:23:00.254651 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.254617    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x98p4" event={"ID":"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d","Type":"ContainerStarted","Data":"985acfab31aca6aede7f97d0918b50246b01874a1306a808732154146be4ceb6"}
Sep 16 21:23:00.262645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.262625    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:23:00.262748 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.262709    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq" event={"ID":"f2378ae2-b1f3-41b3-a99c-1aa20b008f68","Type":"ContainerStarted","Data":"308e418a079d50113ec3da3f171c0a911343ca5980d5beb77d132bed7850aa7e"}
Sep 16 21:23:00.263142 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.263122    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:23:00.264545 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.264523    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx" event={"ID":"8f40f29c-8841-4a0b-8b1a-343e91436f77","Type":"ContainerStarted","Data":"1a438e99c09f2345dbfdeab8c600dc17032095f71877c8766e420a84d56bcf91"}
Sep 16 21:23:00.265917 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.265867    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dh4v8" event={"ID":"38c2151a-8d75-4772-a0a2-922fb7ca7e20","Type":"ContainerStarted","Data":"34b213348675e12b69a3122fac609ccb8eee804d9e357081389db0cd794c9c03"}
Sep 16 21:23:00.267093 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.267070    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794" event={"ID":"f980eff7-4c6c-43da-b2e6-f62f01bda7ee","Type":"ContainerStarted","Data":"8ad8983fc1f073bddac34453ac06aa8f359fe75d07cd618c6d8cbe48d4f1b5cc"}
Sep 16 21:23:00.268220 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.268190    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz" event={"ID":"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14","Type":"ContainerStarted","Data":"b733b5dfc7436989e39f696f117a9249bf39d15fedd149aa832e416626341feb"}
Sep 16 21:23:00.269823 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.269800    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f57db6868-lgg22" event={"ID":"863ab98a-8183-4e24-aba7-8d57e12368c6","Type":"ContainerStarted","Data":"8d3d1c60d601e9fe9e17878da3b78758e700f0b10e5a872fa20a34d5429e95c9"}
Sep 16 21:23:00.269939 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.269835    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5f57db6868-lgg22" event={"ID":"863ab98a-8183-4e24-aba7-8d57e12368c6","Type":"ContainerStarted","Data":"46aa0a669c079a5eb179073b010ca0deb2f3db500b2214bc69a14ec866961315"}
Sep 16 21:23:00.271434 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.271412    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-frkfh" event={"ID":"d4a729ab-f653-4c7c-a0bb-23569feae959","Type":"ContainerStarted","Data":"4ffd3af0678c8d70a1de7fc8b797f94725aff3aed0a7d547a17f4d2deeb491b0"}
Sep 16 21:23:00.271516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.271452    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-frkfh" event={"ID":"d4a729ab-f653-4c7c-a0bb-23569feae959","Type":"ContainerStarted","Data":"982efebf802b2b2d03bb4f72a1f76bcd7529f59fdff33638be31b5cfc979c439"}
Sep 16 21:23:00.271516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.271478    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:23:00.273083 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.273063    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f97687479-dkdrq" event={"ID":"9c2b47d0-5a18-4998-824f-e8d425e414c3","Type":"ContainerStarted","Data":"d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3"}
Sep 16 21:23:00.273163 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.273104    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f97687479-dkdrq" event={"ID":"9c2b47d0-5a18-4998-824f-e8d425e414c3","Type":"ContainerStarted","Data":"7c887945c391cd069c21451dd5222f1e0214ac4ace3326b5a10ff9b40cd6c021"}
Sep 16 21:23:00.282750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.282711    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq" podStartSLOduration=28.410924624 podStartE2EDuration="38.282699097s" totalImagesPullingTime="9.871774473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:00.282516244 +0000 UTC m=+66.930928233" watchObservedRunningTime="2026-09-16 21:23:00.282699097 +0000 UTC m=+66.931111079"
Sep 16 21:23:00.299863 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.299816    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-frkfh" podStartSLOduration=66.299799484 podStartE2EDuration="1m6.299799484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:00.299697089 +0000 UTC m=+66.948109076" watchObservedRunningTime="2026-09-16 21:23:00.299799484 +0000 UTC m=+66.948211473"
Sep 16 21:23:00.316036 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.315995    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-ndwmx" podStartSLOduration=18.164086007 podStartE2EDuration="19.315981319s" totalImagesPullingTime="1.151895312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:00.314929856 +0000 UTC m=+66.963341845" watchObservedRunningTime="2026-09-16 21:23:00.315981319 +0000 UTC m=+66.964393323"
Sep 16 21:23:00.333535 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.333455    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f97687479-dkdrq" podStartSLOduration=44.333442952 podStartE2EDuration="44.333442952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:00.332825327 +0000 UTC m=+66.981237316" watchObservedRunningTime="2026-09-16 21:23:00.333442952 +0000 UTC m=+66.981854940"
Sep 16 21:23:00.352122 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.352062    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5f57db6868-lgg22" podStartSLOduration=38.352042926 podStartE2EDuration="38.352042926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:00.351337527 +0000 UTC m=+66.999749514" watchObservedRunningTime="2026-09-16 21:23:00.352042926 +0000 UTC m=+67.000454915"
Sep 16 21:23:00.810587 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.810548    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:23:00.813510 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.813483    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:23:00.982506 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:00.982472    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-8p9nq"
Sep 16 21:23:01.008730 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.008692    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:01.109266 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.109086    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:01.255674 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.255520    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:01.280825 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.279685    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:23:01.282076 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.281188    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:23:01.283551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.283530    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5f57db6868-lgg22"
Sep 16 21:23:01.340542 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.340480    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:01.565037 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.564945    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:01.697316 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.697268    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:01.790501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.790454    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:01.890039 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:01.890001    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:02.110936 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:02.110835    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:02.210697 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:02.210661    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:02.285694 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:02.285660    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x98p4" event={"ID":"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d","Type":"ContainerStarted","Data":"f770445ad8c2cb93fc5546babd4d8a1b6562f4ae5963d340d703a0bf7b1161e8"}
Sep 16 21:23:02.928022 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:02.927953    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:03.083145 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:03.083104    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:03.423172 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:03.423126    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:04.100143 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.100114    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:04.201753 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.201714    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:04.649776 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.649487    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"]
Sep 16 21:23:04.671537 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.671505    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"]
Sep 16 21:23:04.671878 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.671814    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"
Sep 16 21:23:04.675277 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.675248    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 21:23:04.675673 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.675648    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 21:23:04.676368 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.676240    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 21:23:04.676492 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.676456    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-g8rqd\""
Sep 16 21:23:04.680494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.680098    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 21:23:04.686291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.685527    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"]
Sep 16 21:23:04.688774 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.687724    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.692266 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.692244    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 21:23:04.692636 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.692616    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 21:23:04.695144 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.694686    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 21:23:04.695144 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.694981    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 21:23:04.706055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.706018    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"]
Sep 16 21:23:04.706235 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.706060    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"]
Sep 16 21:23:04.706235 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.706081    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"]
Sep 16 21:23:04.706235 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.706107    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.710291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.710253    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 16 21:23:04.729367 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.729338    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-zjhk6"]
Sep 16 21:23:04.746075 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.746041    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zjhk6"
Sep 16 21:23:04.748453 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.748427    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jd2gd\""
Sep 16 21:23:04.748611 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.748438    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 21:23:04.748683 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.748666    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 21:23:04.755153 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.755131    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zjhk6"]
Sep 16 21:23:04.790232 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790192    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78e6c7a9-2175-47a8-9489-20de6cabecb6-tmp\") pod \"klusterlet-addon-workmgr-67ff889bfc-scfkv\" (UID: \"78e6c7a9-2175-47a8-9489-20de6cabecb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.790232 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790235    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-hub\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.790473 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790263    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.790473 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790345    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b44ba96-5daa-4a7d-9d61-f2203aafcf17-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6766869874-bltcj\" (UID: \"5b44ba96-5daa-4a7d-9d61-f2203aafcf17\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"
Sep 16 21:23:04.790473 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790418    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c458v\" (UniqueName: \"kubernetes.io/projected/78e6c7a9-2175-47a8-9489-20de6cabecb6-kube-api-access-c458v\") pod \"klusterlet-addon-workmgr-67ff889bfc-scfkv\" (UID: \"78e6c7a9-2175-47a8-9489-20de6cabecb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.790473 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790453    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8k4\" (UniqueName: \"kubernetes.io/projected/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-kube-api-access-dx8k4\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.790685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790485    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.790685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790545    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrzpg\" (UniqueName: \"kubernetes.io/projected/5b44ba96-5daa-4a7d-9d61-f2203aafcf17-kube-api-access-mrzpg\") pod \"managed-serviceaccount-addon-agent-6766869874-bltcj\" (UID: \"5b44ba96-5daa-4a7d-9d61-f2203aafcf17\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"
Sep 16 21:23:04.790685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790597    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-ca\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.790685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790621    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.790853 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.790689    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78e6c7a9-2175-47a8-9489-20de6cabecb6-klusterlet-config\") pod \"klusterlet-addon-workmgr-67ff889bfc-scfkv\" (UID: \"78e6c7a9-2175-47a8-9489-20de6cabecb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.829809 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.829759    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:04.891606 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891568    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78e6c7a9-2175-47a8-9489-20de6cabecb6-tmp\") pod \"klusterlet-addon-workmgr-67ff889bfc-scfkv\" (UID: \"78e6c7a9-2175-47a8-9489-20de6cabecb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.891777 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891622    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-hub\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.891777 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891646    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.891777 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891671    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b44ba96-5daa-4a7d-9d61-f2203aafcf17-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6766869874-bltcj\" (UID: \"5b44ba96-5daa-4a7d-9d61-f2203aafcf17\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"
Sep 16 21:23:04.891777 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891710    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c458v\" (UniqueName: \"kubernetes.io/projected/78e6c7a9-2175-47a8-9489-20de6cabecb6-kube-api-access-c458v\") pod \"klusterlet-addon-workmgr-67ff889bfc-scfkv\" (UID: \"78e6c7a9-2175-47a8-9489-20de6cabecb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.891777 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891756    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8k4\" (UniqueName: \"kubernetes.io/projected/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-kube-api-access-dx8k4\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.892056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891788    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.892056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891815    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrzpg\" (UniqueName: \"kubernetes.io/projected/5b44ba96-5daa-4a7d-9d61-f2203aafcf17-kube-api-access-mrzpg\") pod \"managed-serviceaccount-addon-agent-6766869874-bltcj\" (UID: \"5b44ba96-5daa-4a7d-9d61-f2203aafcf17\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"
Sep 16 21:23:04.892056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891857    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-ca\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.892056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891880    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.892056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.891939    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ht6\" (UniqueName: \"kubernetes.io/projected/743eaf68-5cad-4048-a581-0c350e2969ce-kube-api-access-w5ht6\") pod \"downloads-76f99584f4-zjhk6\" (UID: \"743eaf68-5cad-4048-a581-0c350e2969ce\") " pod="openshift-console/downloads-76f99584f4-zjhk6"
Sep 16 21:23:04.892056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.892013    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78e6c7a9-2175-47a8-9489-20de6cabecb6-klusterlet-config\") pod \"klusterlet-addon-workmgr-67ff889bfc-scfkv\" (UID: \"78e6c7a9-2175-47a8-9489-20de6cabecb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.892801 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.892748    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.892952 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.892878    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78e6c7a9-2175-47a8-9489-20de6cabecb6-tmp\") pod \"klusterlet-addon-workmgr-67ff889bfc-scfkv\" (UID: \"78e6c7a9-2175-47a8-9489-20de6cabecb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.895388 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.895347    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5b44ba96-5daa-4a7d-9d61-f2203aafcf17-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6766869874-bltcj\" (UID: \"5b44ba96-5daa-4a7d-9d61-f2203aafcf17\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"
Sep 16 21:23:04.896568 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.895390    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-hub\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.896568 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.896323    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.896723 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.896651    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/78e6c7a9-2175-47a8-9489-20de6cabecb6-klusterlet-config\") pod \"klusterlet-addon-workmgr-67ff889bfc-scfkv\" (UID: \"78e6c7a9-2175-47a8-9489-20de6cabecb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.896858 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.896836    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-ca\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.897151 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.897129    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.909838 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.909756    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8k4\" (UniqueName: \"kubernetes.io/projected/ec730a4a-2974-4dba-a4e0-ee3fe79daba4-kube-api-access-dx8k4\") pod \"cluster-proxy-proxy-agent-78f5488958-7bg7f\" (UID: \"ec730a4a-2974-4dba-a4e0-ee3fe79daba4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:04.910131 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.910111    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrzpg\" (UniqueName: \"kubernetes.io/projected/5b44ba96-5daa-4a7d-9d61-f2203aafcf17-kube-api-access-mrzpg\") pod \"managed-serviceaccount-addon-agent-6766869874-bltcj\" (UID: \"5b44ba96-5daa-4a7d-9d61-f2203aafcf17\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"
Sep 16 21:23:04.910239 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.910144    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c458v\" (UniqueName: \"kubernetes.io/projected/78e6c7a9-2175-47a8-9489-20de6cabecb6-kube-api-access-c458v\") pod \"klusterlet-addon-workmgr-67ff889bfc-scfkv\" (UID: \"78e6c7a9-2175-47a8-9489-20de6cabecb6\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:04.993250 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.993213    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"
Sep 16 21:23:04.993457 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:04.993430    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ht6\" (UniqueName: \"kubernetes.io/projected/743eaf68-5cad-4048-a581-0c350e2969ce-kube-api-access-w5ht6\") pod \"downloads-76f99584f4-zjhk6\" (UID: \"743eaf68-5cad-4048-a581-0c350e2969ce\") " pod="openshift-console/downloads-76f99584f4-zjhk6"
Sep 16 21:23:05.004702 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.004669    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"
Sep 16 21:23:05.011321 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.011296    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ht6\" (UniqueName: \"kubernetes.io/projected/743eaf68-5cad-4048-a581-0c350e2969ce-kube-api-access-w5ht6\") pod \"downloads-76f99584f4-zjhk6\" (UID: \"743eaf68-5cad-4048-a581-0c350e2969ce\") " pod="openshift-console/downloads-76f99584f4-zjhk6"
Sep 16 21:23:05.023884 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.023845    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:05.026648 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.026615    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:05.058063 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.058030    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-zjhk6"
Sep 16 21:23:05.119834 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.119799    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:05.223516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.223421    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:05.300240 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.300198    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:05.371054 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.371016    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:05.866004 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.865845    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj"]
Sep 16 21:23:05.882765 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.882720    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-zjhk6"]
Sep 16 21:23:05.890111 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.890088    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f"]
Sep 16 21:23:05.903106 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:05.903061    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b44ba96_5daa_4a7d_9d61_f2203aafcf17.slice/crio-d56a5930fb46cf22def3ff85bac0461d95592d7c00f20e1f29d6fbf5c0936a26 WatchSource:0}: Error finding container d56a5930fb46cf22def3ff85bac0461d95592d7c00f20e1f29d6fbf5c0936a26: Status 404 returned error can't find the container with id d56a5930fb46cf22def3ff85bac0461d95592d7c00f20e1f29d6fbf5c0936a26
Sep 16 21:23:05.904021 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:05.903998    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"]
Sep 16 21:23:05.905649 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:05.905588    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743eaf68_5cad_4048_a581_0c350e2969ce.slice/crio-07446f99e567af7d81f33221e6e251e991ea39d773581978fddcb63da7ef41db WatchSource:0}: Error finding container 07446f99e567af7d81f33221e6e251e991ea39d773581978fddcb63da7ef41db: Status 404 returned error can't find the container with id 07446f99e567af7d81f33221e6e251e991ea39d773581978fddcb63da7ef41db
Sep 16 21:23:05.905875 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:05.905822    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec730a4a_2974_4dba_a4e0_ee3fe79daba4.slice/crio-2322ea3ca413b1b7023fbe77e30d573f63deaf8c67d4482c830b26191fd2adc0 WatchSource:0}: Error finding container 2322ea3ca413b1b7023fbe77e30d573f63deaf8c67d4482c830b26191fd2adc0: Status 404 returned error can't find the container with id 2322ea3ca413b1b7023fbe77e30d573f63deaf8c67d4482c830b26191fd2adc0
Sep 16 21:23:05.908230 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:05.908200    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e6c7a9_2175_47a8_9489_20de6cabecb6.slice/crio-88511d5ffbe05fd75a11d3088adf7b9de1e15103cf8afd131e76c4bea0326910 WatchSource:0}: Error finding container 88511d5ffbe05fd75a11d3088adf7b9de1e15103cf8afd131e76c4bea0326910: Status 404 returned error can't find the container with id 88511d5ffbe05fd75a11d3088adf7b9de1e15103cf8afd131e76c4bea0326910
Sep 16 21:23:06.121003 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.120888    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:06.223745 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.223704    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:06.279648 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.279567    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"]
Sep 16 21:23:06.301306 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.301275    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"]
Sep 16 21:23:06.301461 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.301379    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"
Sep 16 21:23:06.304205 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.304182    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bql2b\""
Sep 16 21:23:06.304344 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.304287    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 21:23:06.311988 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.311954    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x98p4" event={"ID":"ffed9181-7dab-40e2-aa2a-94f3ef7dae5d","Type":"ContainerStarted","Data":"1382e5d640bd22657b55f2c45d8d88d4c13fa4bab5d8f4763df321e339e4e1c0"}
Sep 16 21:23:06.313314 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.313286    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv" event={"ID":"78e6c7a9-2175-47a8-9489-20de6cabecb6","Type":"ContainerStarted","Data":"88511d5ffbe05fd75a11d3088adf7b9de1e15103cf8afd131e76c4bea0326910"}
Sep 16 21:23:06.314873 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.314850    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dh4v8" event={"ID":"38c2151a-8d75-4772-a0a2-922fb7ca7e20","Type":"ContainerStarted","Data":"ea422ad2bafbb70d09a17e505d8ba9b1e88ba17468e0b1db400583f923b32cbd"}
Sep 16 21:23:06.316128 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.316108    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zjhk6" event={"ID":"743eaf68-5cad-4048-a581-0c350e2969ce","Type":"ContainerStarted","Data":"07446f99e567af7d81f33221e6e251e991ea39d773581978fddcb63da7ef41db"}
Sep 16 21:23:06.317883 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.317862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794" event={"ID":"f980eff7-4c6c-43da-b2e6-f62f01bda7ee","Type":"ContainerStarted","Data":"ca1cfcc836b8513a4fa2b7ac6d3ec5825446cef36acfbf79887378037110ed89"}
Sep 16 21:23:06.317999 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.317922    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794" event={"ID":"f980eff7-4c6c-43da-b2e6-f62f01bda7ee","Type":"ContainerStarted","Data":"9cf44b90fbfcb2d516f47b7b9596b550a2a4e11a5175616de53cdec72cea3856"}
Sep 16 21:23:06.319437 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.319416    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz" event={"ID":"2cfc325f-85d2-4a82-aa0e-12c2ff6beb14","Type":"ContainerStarted","Data":"8e35c913642cbd6a2dae488197dc30490b98af7e79a04fd08a1c25cb7332abcb"}
Sep 16 21:23:06.320631 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.320613    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f" event={"ID":"ec730a4a-2974-4dba-a4e0-ee3fe79daba4","Type":"ContainerStarted","Data":"2322ea3ca413b1b7023fbe77e30d573f63deaf8c67d4482c830b26191fd2adc0"}
Sep 16 21:23:06.321739 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.321718    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj" event={"ID":"5b44ba96-5daa-4a7d-9d61-f2203aafcf17","Type":"ContainerStarted","Data":"d56a5930fb46cf22def3ff85bac0461d95592d7c00f20e1f29d6fbf5c0936a26"}
Sep 16 21:23:06.323553 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.323532    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45xsc" event={"ID":"1e8049ff-a69e-4607-b24f-b98b40f53fc2","Type":"ContainerStarted","Data":"d2ab6c265289ab5b0d09035d4c06d5ba8d41a1242844b32db4c63889b036802b"}
Sep 16 21:23:06.323637 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.323622    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45xsc" event={"ID":"1e8049ff-a69e-4607-b24f-b98b40f53fc2","Type":"ContainerStarted","Data":"686b0501e48841cfab9bfaa516c20eef25f0f286fe86539af025364018cc7b4b"}
Sep 16 21:23:06.323711 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.323655    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-45xsc"
Sep 16 21:23:06.339261 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.339224    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:06.369969 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.369881    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-45xsc" podStartSLOduration=34.006296942 podStartE2EDuration="39.369843358s" totalImagesPullingTime="5.363546416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:06.369330406 +0000 UTC m=+73.017742395" watchObservedRunningTime="2026-09-16 21:23:06.369843358 +0000 UTC m=+73.018255345"
Sep 16 21:23:06.406774 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.406737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ce2ec2d-3586-41bc-8db4-a15671fe8c4d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7jmwp\" (UID: \"0ce2ec2d-3586-41bc-8db4-a15671fe8c4d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"
Sep 16 21:23:06.410121 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.409983    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-rb794" podStartSLOduration=39.075735049 podStartE2EDuration="44.409966544s" totalImagesPullingTime="5.334231495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:06.409949105 +0000 UTC m=+73.058361092" watchObservedRunningTime="2026-09-16 21:23:06.409966544 +0000 UTC m=+73.058378529"
Sep 16 21:23:06.432625 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.432568    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-5ldzz" podStartSLOduration=39.007459137 podStartE2EDuration="44.432549405s" totalImagesPullingTime="5.425090268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:22:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:06.43148306 +0000 UTC m=+73.079895049" watchObservedRunningTime="2026-09-16 21:23:06.432549405 +0000 UTC m=+73.080961397"
Sep 16 21:23:06.448869 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.448821    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dh4v8" podStartSLOduration=66.586449422 podStartE2EDuration="1m12.448805991s" totalImagesPullingTime="5.862356569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:06.447739006 +0000 UTC m=+73.096150984" watchObservedRunningTime="2026-09-16 21:23:06.448805991 +0000 UTC m=+73.097217979"
Sep 16 21:23:06.462218 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.462187    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:06.469090 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.469042    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x98p4" podStartSLOduration=70.792829572 podStartE2EDuration="1m12.469027703s" totalImagesPullingTime="1.676198131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:06.467966003 +0000 UTC m=+73.116377990" watchObservedRunningTime="2026-09-16 21:23:06.469027703 +0000 UTC m=+73.117439690"
Sep 16 21:23:06.507814 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.507768    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ce2ec2d-3586-41bc-8db4-a15671fe8c4d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7jmwp\" (UID: \"0ce2ec2d-3586-41bc-8db4-a15671fe8c4d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"
Sep 16 21:23:06.510285 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.510257    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/0ce2ec2d-3586-41bc-8db4-a15671fe8c4d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7jmwp\" (UID: \"0ce2ec2d-3586-41bc-8db4-a15671fe8c4d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"
Sep 16 21:23:06.536530 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.536417    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-865fb5976-94qtx"]
Sep 16 21:23:06.559581 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.559543    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865fb5976-94qtx"]
Sep 16 21:23:06.559748 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.559689    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.562187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.562158    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 21:23:06.562333 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.562211    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 21:23:06.562333 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.562159    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 21:23:06.562433 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.562334    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 21:23:06.562433 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.562417    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 21:23:06.562598 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.562581    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vpfpz\""
Sep 16 21:23:06.613032 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.613000    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"
Sep 16 21:23:06.626440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.626405    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:06.711067 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.711032    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-console-config\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.711220 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.711091    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5h2h\" (UniqueName: \"kubernetes.io/projected/c90ad643-4367-4547-80b1-d72f7e9f3347-kube-api-access-h5h2h\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.711220 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.711145    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-oauth-config\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.711220 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.711195    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-serving-cert\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.711386 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.711237    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-oauth-serving-cert\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.711386 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.711273    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-service-ca\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.713298 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.713246    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:06.813281 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.812615    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-oauth-config\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.813281 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.813230    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-serving-cert\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.813491 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.813314    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-oauth-serving-cert\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.813491 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.813356    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-service-ca\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.813491 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.813413    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-console-config\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.813491 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.813450    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5h2h\" (UniqueName: \"kubernetes.io/projected/c90ad643-4367-4547-80b1-d72f7e9f3347-kube-api-access-h5h2h\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.814833 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.814491    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-console-config\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.814833 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.814526    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-service-ca\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.814833 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.814785    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-oauth-serving-cert\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.817438 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.817414    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-oauth-config\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.817563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.817445    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-serving-cert\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.824153 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.824129    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5h2h\" (UniqueName: \"kubernetes.io/projected/c90ad643-4367-4547-80b1-d72f7e9f3347-kube-api-access-h5h2h\") pod \"console-865fb5976-94qtx\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") " pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.871298 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.871260    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:06.916486 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:06.916451    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"]
Sep 16 21:23:06.922941 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:06.922873    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce2ec2d_3586_41bc_8db4_a15671fe8c4d.slice/crio-6de5b4357baca5f3276fd7d45c875a587c60bdd255677dec06696373b8c323ee WatchSource:0}: Error finding container 6de5b4357baca5f3276fd7d45c875a587c60bdd255677dec06696373b8c323ee: Status 404 returned error can't find the container with id 6de5b4357baca5f3276fd7d45c875a587c60bdd255677dec06696373b8c323ee
Sep 16 21:23:07.215847 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:07.215510    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865fb5976-94qtx"]
Sep 16 21:23:07.225564 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:23:07.220396    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90ad643_4367_4547_80b1_d72f7e9f3347.slice/crio-71f3dd4af4d48b2fd61c0c3391da81521856b787d47e5786816ed2db6ce495c4 WatchSource:0}: Error finding container 71f3dd4af4d48b2fd61c0c3391da81521856b787d47e5786816ed2db6ce495c4: Status 404 returned error can't find the container with id 71f3dd4af4d48b2fd61c0c3391da81521856b787d47e5786816ed2db6ce495c4
Sep 16 21:23:07.337028 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:07.336929    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp" event={"ID":"0ce2ec2d-3586-41bc-8db4-a15671fe8c4d","Type":"ContainerStarted","Data":"6de5b4357baca5f3276fd7d45c875a587c60bdd255677dec06696373b8c323ee"}
Sep 16 21:23:07.341660 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:07.341593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865fb5976-94qtx" event={"ID":"c90ad643-4367-4547-80b1-d72f7e9f3347","Type":"ContainerStarted","Data":"71f3dd4af4d48b2fd61c0c3391da81521856b787d47e5786816ed2db6ce495c4"}
Sep 16 21:23:07.413069 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:07.413018    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:10.105010 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:10.104848    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:10.242418 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:10.242378    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:14.021328 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:14.021283    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:14.123186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:14.123153    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:15.530358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:15.530319    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:15.614333 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:15.614288    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:15.693925 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:15.693877    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:16.345715 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.345686    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-45xsc"
Sep 16 21:23:16.403199 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.403167    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv" event={"ID":"78e6c7a9-2175-47a8-9489-20de6cabecb6","Type":"ContainerStarted","Data":"0a794af134a654b410b4098a8da5c66f09a93aefdea2ef014748666241955804"}
Sep 16 21:23:16.403654 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.403624    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:16.405379 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.405354    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv"
Sep 16 21:23:16.406383 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.406354    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f" event={"ID":"ec730a4a-2974-4dba-a4e0-ee3fe79daba4","Type":"ContainerStarted","Data":"b68edbb4b685b4a2d06d84e6f31ec6c727c00f44d7dda79d42e66734f2958af9"}
Sep 16 21:23:16.406500 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.406396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f" event={"ID":"ec730a4a-2974-4dba-a4e0-ee3fe79daba4","Type":"ContainerStarted","Data":"c207390c88a32103d5baf87bf4250c08dcc08d7356aa09d6f14d21a696968628"}
Sep 16 21:23:16.406500 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.406415    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f" event={"ID":"ec730a4a-2974-4dba-a4e0-ee3fe79daba4","Type":"ContainerStarted","Data":"3fd6d87b5fd0de379b00cbee81fdaffd848c235de52a9e289f5e120e5da0afec"}
Sep 16 21:23:16.408564 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.408537    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj" event={"ID":"5b44ba96-5daa-4a7d-9d61-f2203aafcf17","Type":"ContainerStarted","Data":"51d227a1c8e8181452df4b08c40fe07f4c100b3ac01d688822f329f3aeabb86e"}
Sep 16 21:23:16.410778 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.410756    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp" event={"ID":"0ce2ec2d-3586-41bc-8db4-a15671fe8c4d","Type":"ContainerStarted","Data":"74c487fefa26ff8958ed8ce1e1159d4021eab0a66f26c9bb2564216d64efa912"}
Sep 16 21:23:16.410884 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.410868    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"
Sep 16 21:23:16.412685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.412665    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865fb5976-94qtx" event={"ID":"c90ad643-4367-4547-80b1-d72f7e9f3347","Type":"ContainerStarted","Data":"4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0"}
Sep 16 21:23:16.415760 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.415736    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp"
Sep 16 21:23:16.420846 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.420808    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67ff889bfc-scfkv" podStartSLOduration=2.6254284329999997 podStartE2EDuration="12.420796667s" totalImagesPullingTime="9.795368234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:16.419887682 +0000 UTC m=+83.068299670" watchObservedRunningTime="2026-09-16 21:23:16.420796667 +0000 UTC m=+83.069208655"
Sep 16 21:23:16.441484 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.441434    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-865fb5976-94qtx" podStartSLOduration=1.900730951 podStartE2EDuration="10.441415238s" totalImagesPullingTime="8.540684287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:16.43907899 +0000 UTC m=+83.087490979" watchObservedRunningTime="2026-09-16 21:23:16.441415238 +0000 UTC m=+83.089827221"
Sep 16 21:23:16.457453 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.457411    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78f5488958-7bg7f" podStartSLOduration=2.662315183 podStartE2EDuration="12.457396863s" totalImagesPullingTime="9.79508168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:16.456927743 +0000 UTC m=+83.105339732" watchObservedRunningTime="2026-09-16 21:23:16.457396863 +0000 UTC m=+83.105808851"
Sep 16 21:23:16.474117 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.474072    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7jmwp" podStartSLOduration=1.685927096 podStartE2EDuration="10.47406145s" totalImagesPullingTime="8.788134354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:16.473325826 +0000 UTC m=+83.121737815" watchObservedRunningTime="2026-09-16 21:23:16.47406145 +0000 UTC m=+83.122473437"
Sep 16 21:23:16.492164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.492116    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6766869874-bltcj" podStartSLOduration=2.697422651 podStartE2EDuration="12.492101164s" totalImagesPullingTime="9.794678513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:16.491456428 +0000 UTC m=+83.139868417" watchObservedRunningTime="2026-09-16 21:23:16.492101164 +0000 UTC m=+83.140513151"
Sep 16 21:23:16.871964 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.871921    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:16.872426 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.871995    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:16.873550 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.873515    2567 patch_prober.go:28] interesting pod/console-865fb5976-94qtx container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.27:8443/health\": dial tcp 10.132.0.27:8443: connect: connection refused" start-of-body=
Sep 16 21:23:16.873712 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:16.873620    2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-865fb5976-94qtx" podUID="c90ad643-4367-4547-80b1-d72f7e9f3347" containerName="console" probeResult="failure" output="Get \"https://10.132.0.27:8443/health\": dial tcp 10.132.0.27:8443: connect: connection refused"
Sep 16 21:23:17.291170 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:17.291131    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:17.392093 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:17.392046    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:17.491529 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:17.491488    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:19.809589 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:19.809553    2567 patch_prober.go:28] interesting pod/image-registry-5f97687479-dkdrq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 21:23:19.810064 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:19.809643    2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5f97687479-dkdrq" podUID="9c2b47d0-5a18-4998-824f-e8d425e414c3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 21:23:20.301494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:20.301452    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:20.376546 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:20.376504    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:20.794728 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:20.794694    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:20.897081 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:20.897038    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:20.999508 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:20.999468    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:21.098138 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:21.098043    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:21.543360 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:21.543305    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:21.643449 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:21.643412    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:21.912689 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:21.912605    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:22.290493 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:22.290460    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:23:22.563630 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:22.563543    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:22.662885 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:22.662809    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:24.420943 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.420892    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:24.653406 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.653369    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:24.724983 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.724886    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cz9qx"]
Sep 16 21:23:24.748473 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.748437    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.751111 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.750612    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 21:23:24.751111 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.750772    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-q9ngt\""
Sep 16 21:23:24.751111 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.750878    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 21:23:24.751350 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.751159    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 21:23:24.751719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.751692    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 21:23:24.768542 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.768508    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:24.777310 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.777285    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-root\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.777449 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.777332    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-sys\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.777449 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.777357    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-textfile\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.777449 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.777401    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrwz\" (UniqueName: \"kubernetes.io/projected/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-kube-api-access-dvrwz\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.777449 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.777431    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-tls\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.777646 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.777461    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.777646 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.777489    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-metrics-client-ca\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.777646 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.777527    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-accelerators-collector-config\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.777790 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.777653    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-wtmp\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.878754 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.878716    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrwz\" (UniqueName: \"kubernetes.io/projected/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-kube-api-access-dvrwz\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.878754 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.878758    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-tls\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879014 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.878785    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879014 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.878812    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-metrics-client-ca\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879014 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.878850    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-accelerators-collector-config\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879014 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:23:24.878878    2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 21:23:24.879014 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:23:24.878969    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-tls podName:fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5 nodeName:}" failed. No retries permitted until 2026-09-16 21:23:25.378947687 +0000 UTC m=+92.027359667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-tls") pod "node-exporter-cz9qx" (UID: "fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5") : secret "node-exporter-tls" not found
Sep 16 21:23:24.879014 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.878879    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-wtmp\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.879027    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-wtmp\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.879042    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-root\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.879092    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-sys\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.879118    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-textfile\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.879306    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-root\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.879322    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-sys\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879635 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.879555    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-metrics-client-ca\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879821 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.879802    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-textfile\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.879946 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.879925    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-accelerators-collector-config\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.881216 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.881194    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.897971 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.897929    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:24.903563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.903529    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrwz\" (UniqueName: \"kubernetes.io/projected/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-kube-api-access-dvrwz\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:24.998422 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:24.998386    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:25.092421 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.092394    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:25.192888 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.192854    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:25.389545 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.389508    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-tls\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:25.391792 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.391766    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5-node-exporter-tls\") pod \"node-exporter-cz9qx\" (UID: \"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5\") " pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:25.454818 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.454778    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-zjhk6" event={"ID":"743eaf68-5cad-4048-a581-0c350e2969ce","Type":"ContainerStarted","Data":"1a58c8976d67b193c0cfac55a8600a024bc370e3de99ccb03cfb39800d68aa04"}
Sep 16 21:23:25.455295 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.454975    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-zjhk6"
Sep 16 21:23:25.456532 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.456490    2567 patch_prober.go:28] interesting pod/downloads-76f99584f4-zjhk6 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" start-of-body=
Sep 16 21:23:25.456830 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.456547    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-zjhk6" podUID="743eaf68-5cad-4048-a581-0c350e2969ce" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused"
Sep 16 21:23:25.474957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.474882    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-zjhk6" podStartSLOduration=2.241491739 podStartE2EDuration="21.474863781s" totalImagesPullingTime="19.233372042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:25.473976768 +0000 UTC m=+92.122388753" watchObservedRunningTime="2026-09-16 21:23:25.474863781 +0000 UTC m=+92.123275771"
Sep 16 21:23:25.504196 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.504155    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:25.606937 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.606828    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:25.660444 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.660245    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cz9qx"
Sep 16 21:23:25.707327 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.707291    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:25.776499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.776446    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:25.978188 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:25.975742    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865fb5976-94qtx"]
Sep 16 21:23:26.136113 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:26.136071    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:26.248835 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:26.248807    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:26.337857 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:26.337241    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:26.469685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:26.469593    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cz9qx" event={"ID":"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5","Type":"ContainerStarted","Data":"7d079fbaba9aea8b90bace0299a3917e0a046829389b01c89d938cc71498d1a2"}
Sep 16 21:23:26.477968 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:26.477855    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-zjhk6"
Sep 16 21:23:27.576384 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:27.576349    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:27.672398 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:27.672248    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:27.780477 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:27.780442    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:27.877145 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:27.877109    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:27.964019 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:27.963921    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:28.128967 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:28.128926    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:28.228301 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:28.228214    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:28.482175 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:28.482088    2567 generic.go:356] "Generic (PLEG): container finished" podID="fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5" containerID="c493e1e7f580442e8c1410f362d1ef3f0677082054b3cc48336c1320dd10724d" exitCode=0
Sep 16 21:23:28.482175 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:28.482138    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cz9qx" event={"ID":"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5","Type":"ContainerDied","Data":"c493e1e7f580442e8c1410f362d1ef3f0677082054b3cc48336c1320dd10724d"}
Sep 16 21:23:28.589252 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:28.589223    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:28.699731 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:28.699648    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:29.198354 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:29.198263    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:29.268611 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:29.268553    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:29.490842 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:29.490763    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cz9qx" event={"ID":"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5","Type":"ContainerStarted","Data":"b254e79a5ec7e8053b28646c75bae5cc31c5b7f03171384c82e92a63c963662c"}
Sep 16 21:23:29.491012 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:29.490855    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cz9qx" event={"ID":"fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5","Type":"ContainerStarted","Data":"06e82ae6dfd7ab34fbcb79a72cf3d1eab9da6a7376ecc34d1431101417201f1e"}
Sep 16 21:23:29.510171 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:29.510137    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:29.539187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:29.539120    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cz9qx" podStartSLOduration=3.769565479 podStartE2EDuration="5.539097741s" totalImagesPullingTime="1.68957853s" totalInitContainerRuntime="79.953732ms" isStatefulPod=true podCreationTimestamp="2026-09-16 21:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:23:29.538107806 +0000 UTC m=+96.186519794" watchObservedRunningTime="2026-09-16 21:23:29.539097741 +0000 UTC m=+96.187509731"
Sep 16 21:23:29.612068 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:29.612028    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:29.720055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:29.720011    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:30.108552 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:30.108512    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:30.188155 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:30.188118    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:30.943068 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:30.943011    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:31.208291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:31.208209    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:31.283977 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:31.283948    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-frkfh"
Sep 16 21:23:31.312168 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:31.312132    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:31.919926 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:31.919880    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f97687479-dkdrq"]
Sep 16 21:23:32.003115 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:32.003074    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:32.096872 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:32.096835    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:34.602160 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:34.602125    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:34.697844 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:34.697807    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:35.298267 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:35.298233    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:35.413767 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:35.413734    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:38.278547 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:38.278508    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:38.381836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:38.381801    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:41.440967 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:41.440930    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:41.502137 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:41.502101    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:43.506412 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:43.506379    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:43.557136 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:43.557083    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:45.247834 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:45.247793    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:45.348371 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:45.348335    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:45.450529 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:45.450495    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:49.654439 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:49.654399    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:49.713008 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:49.712965    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:50.292518 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:50.292476    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:50.351038 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:50.351007    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:23:51.006188 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.006150    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-865fb5976-94qtx" podUID="c90ad643-4367-4547-80b1-d72f7e9f3347" containerName="console" containerID="cri-o://4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0" gracePeriod=15
Sep 16 21:23:51.290075 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.290047    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:51.420685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.420645    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-serving-cert\") pod \"c90ad643-4367-4547-80b1-d72f7e9f3347\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") "
Sep 16 21:23:51.420685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.420690    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-console-config\") pod \"c90ad643-4367-4547-80b1-d72f7e9f3347\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") "
Sep 16 21:23:51.420972 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.420738    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-service-ca\") pod \"c90ad643-4367-4547-80b1-d72f7e9f3347\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") "
Sep 16 21:23:51.420972 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.420781    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-oauth-config\") pod \"c90ad643-4367-4547-80b1-d72f7e9f3347\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") "
Sep 16 21:23:51.420972 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.420804    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5h2h\" (UniqueName: \"kubernetes.io/projected/c90ad643-4367-4547-80b1-d72f7e9f3347-kube-api-access-h5h2h\") pod \"c90ad643-4367-4547-80b1-d72f7e9f3347\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") "
Sep 16 21:23:51.420972 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.420824    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-oauth-serving-cert\") pod \"c90ad643-4367-4547-80b1-d72f7e9f3347\" (UID: \"c90ad643-4367-4547-80b1-d72f7e9f3347\") "
Sep 16 21:23:51.421278 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.421242    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-console-config" (OuterVolumeSpecName: "console-config") pod "c90ad643-4367-4547-80b1-d72f7e9f3347" (UID: "c90ad643-4367-4547-80b1-d72f7e9f3347"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:23:51.421278 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.421258    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-service-ca" (OuterVolumeSpecName: "service-ca") pod "c90ad643-4367-4547-80b1-d72f7e9f3347" (UID: "c90ad643-4367-4547-80b1-d72f7e9f3347"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:23:51.421387 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.421335    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c90ad643-4367-4547-80b1-d72f7e9f3347" (UID: "c90ad643-4367-4547-80b1-d72f7e9f3347"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:23:51.423370 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.423343    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c90ad643-4367-4547-80b1-d72f7e9f3347" (UID: "c90ad643-4367-4547-80b1-d72f7e9f3347"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:23:51.423370 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.423352    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90ad643-4367-4547-80b1-d72f7e9f3347-kube-api-access-h5h2h" (OuterVolumeSpecName: "kube-api-access-h5h2h") pod "c90ad643-4367-4547-80b1-d72f7e9f3347" (UID: "c90ad643-4367-4547-80b1-d72f7e9f3347"). InnerVolumeSpecName "kube-api-access-h5h2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:23:51.423492 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.423376    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c90ad643-4367-4547-80b1-d72f7e9f3347" (UID: "c90ad643-4367-4547-80b1-d72f7e9f3347"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:23:51.521794 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.521700    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5h2h\" (UniqueName: \"kubernetes.io/projected/c90ad643-4367-4547-80b1-d72f7e9f3347-kube-api-access-h5h2h\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:51.521794 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.521735    2567 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-oauth-serving-cert\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:51.521794 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.521745    2567 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-serving-cert\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:51.521794 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.521753    2567 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-console-config\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:51.521794 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.521763    2567 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c90ad643-4367-4547-80b1-d72f7e9f3347-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:51.521794 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.521772    2567 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c90ad643-4367-4547-80b1-d72f7e9f3347-console-oauth-config\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:51.593657 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.593618    2567 generic.go:356] "Generic (PLEG): container finished" podID="c90ad643-4367-4547-80b1-d72f7e9f3347" containerID="4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0" exitCode=0
Sep 16 21:23:51.593837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.593667    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865fb5976-94qtx"
Sep 16 21:23:51.593837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.593709    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865fb5976-94qtx" event={"ID":"c90ad643-4367-4547-80b1-d72f7e9f3347","Type":"ContainerDied","Data":"4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0"}
Sep 16 21:23:51.593837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.593775    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865fb5976-94qtx" event={"ID":"c90ad643-4367-4547-80b1-d72f7e9f3347","Type":"ContainerDied","Data":"71f3dd4af4d48b2fd61c0c3391da81521856b787d47e5786816ed2db6ce495c4"}
Sep 16 21:23:51.593837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.593805    2567 scope.go:117] "RemoveContainer" containerID="4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0"
Sep 16 21:23:51.603216 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.603197    2567 scope.go:117] "RemoveContainer" containerID="4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0"
Sep 16 21:23:51.603578 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:23:51.603554    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0\": container with ID starting with 4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0 not found: ID does not exist" containerID="4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0"
Sep 16 21:23:51.603647 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.603593    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0"} err="failed to get container status \"4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0\": rpc error: code = NotFound desc = could not find container \"4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0\": container with ID starting with 4b4e28a2d01d492b77df38c56c2ba528ad127e13fcd4169689860119b6e19ec0 not found: ID does not exist"
Sep 16 21:23:51.614608 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.614582    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865fb5976-94qtx"]
Sep 16 21:23:51.616434 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.616412    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-865fb5976-94qtx"]
Sep 16 21:23:51.940680 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:51.940605    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c90ad643-4367-4547-80b1-d72f7e9f3347" path="/var/lib/kubelet/pods/c90ad643-4367-4547-80b1-d72f7e9f3347/volumes"
Sep 16 21:23:54.608607 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:54.608571    2567 generic.go:356] "Generic (PLEG): container finished" podID="2908a138-2d0e-4eb8-8b35-742e861fb491" containerID="8bd26a8f2491a1730c0bf173afdf9cce165124f0522a3556e3dde12d146e9457" exitCode=0
Sep 16 21:23:54.609041 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:54.608645    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs" event={"ID":"2908a138-2d0e-4eb8-8b35-742e861fb491","Type":"ContainerDied","Data":"8bd26a8f2491a1730c0bf173afdf9cce165124f0522a3556e3dde12d146e9457"}
Sep 16 21:23:54.609093 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:54.609067    2567 scope.go:117] "RemoveContainer" containerID="8bd26a8f2491a1730c0bf173afdf9cce165124f0522a3556e3dde12d146e9457"
Sep 16 21:23:54.610244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:54.610215    2567 generic.go:356] "Generic (PLEG): container finished" podID="9f518039-df50-4fbb-9d28-f6e554a1a784" containerID="1940bf4f958a179f7f09b3468a67d46e2715ce29c8d5e91cab114f5458f95afc" exitCode=0
Sep 16 21:23:54.610365 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:54.610286    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s2szh" event={"ID":"9f518039-df50-4fbb-9d28-f6e554a1a784","Type":"ContainerDied","Data":"1940bf4f958a179f7f09b3468a67d46e2715ce29c8d5e91cab114f5458f95afc"}
Sep 16 21:23:54.610592 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:54.610571    2567 scope.go:117] "RemoveContainer" containerID="1940bf4f958a179f7f09b3468a67d46e2715ce29c8d5e91cab114f5458f95afc"
Sep 16 21:23:54.611751 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:54.611730    2567 generic.go:356] "Generic (PLEG): container finished" podID="5258e8e0-274b-4247-966f-99e56319eb18" containerID="3794a1545686fa37a7b2a2deb04b6bbf98ee4f0067b37cde02041eddf9c33bf2" exitCode=0
Sep 16 21:23:54.611830 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:54.611798    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv" event={"ID":"5258e8e0-274b-4247-966f-99e56319eb18","Type":"ContainerDied","Data":"3794a1545686fa37a7b2a2deb04b6bbf98ee4f0067b37cde02041eddf9c33bf2"}
Sep 16 21:23:54.612116 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:54.612100    2567 scope.go:117] "RemoveContainer" containerID="3794a1545686fa37a7b2a2deb04b6bbf98ee4f0067b37cde02041eddf9c33bf2"
Sep 16 21:23:55.617966 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:55.617927    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-g9pqs" event={"ID":"2908a138-2d0e-4eb8-8b35-742e861fb491","Type":"ContainerStarted","Data":"23ba43d5a5a522fa07a37a42f001f06c23cb09a8be4b2dd7a1d5f9a16ec74a87"}
Sep 16 21:23:55.619885 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:55.619862    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-s2szh" event={"ID":"9f518039-df50-4fbb-9d28-f6e554a1a784","Type":"ContainerStarted","Data":"86fd5d0a052dbd97479bcc1cccb6eaa71172a9999b400230de4efb75d572fb81"}
Sep 16 21:23:55.621727 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:55.621708    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-hf4dv" event={"ID":"5258e8e0-274b-4247-966f-99e56319eb18","Type":"ContainerStarted","Data":"372d3c7cff6c0de95691586b349095b1aae87904875dce7dd407f4a2bd37701b"}
Sep 16 21:23:56.942779 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:56.942709    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f97687479-dkdrq" podUID="9c2b47d0-5a18-4998-824f-e8d425e414c3" containerName="registry" containerID="cri-o://d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3" gracePeriod=30
Sep 16 21:23:57.212111 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.212029    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:23:57.368407 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.368374    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c2b47d0-5a18-4998-824f-e8d425e414c3-ca-trust-extracted\") pod \"9c2b47d0-5a18-4998-824f-e8d425e414c3\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") "
Sep 16 21:23:57.368578 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.368452    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-trusted-ca\") pod \"9c2b47d0-5a18-4998-824f-e8d425e414c3\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") "
Sep 16 21:23:57.368578 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.368484    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-certificates\") pod \"9c2b47d0-5a18-4998-824f-e8d425e414c3\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") "
Sep 16 21:23:57.368578 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.368517    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-bound-sa-token\") pod \"9c2b47d0-5a18-4998-824f-e8d425e414c3\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") "
Sep 16 21:23:57.368578 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.368541    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") pod \"9c2b47d0-5a18-4998-824f-e8d425e414c3\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") "
Sep 16 21:23:57.368791 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.368696    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-installation-pull-secrets\") pod \"9c2b47d0-5a18-4998-824f-e8d425e414c3\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") "
Sep 16 21:23:57.368849 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.368789    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj8nd\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-kube-api-access-qj8nd\") pod \"9c2b47d0-5a18-4998-824f-e8d425e414c3\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") "
Sep 16 21:23:57.368849 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.368823    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-image-registry-private-configuration\") pod \"9c2b47d0-5a18-4998-824f-e8d425e414c3\" (UID: \"9c2b47d0-5a18-4998-824f-e8d425e414c3\") "
Sep 16 21:23:57.368985 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.368962    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9c2b47d0-5a18-4998-824f-e8d425e414c3" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:23:57.369053 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.369003    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9c2b47d0-5a18-4998-824f-e8d425e414c3" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:23:57.369114 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.369099    2567 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-trusted-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:57.369166 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.369117    2567 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-certificates\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:57.371181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.371148    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9c2b47d0-5a18-4998-824f-e8d425e414c3" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:23:57.371293 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.371159    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9c2b47d0-5a18-4998-824f-e8d425e414c3" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:23:57.371451 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.371424    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9c2b47d0-5a18-4998-824f-e8d425e414c3" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:23:57.371537 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.371463    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-kube-api-access-qj8nd" (OuterVolumeSpecName: "kube-api-access-qj8nd") pod "9c2b47d0-5a18-4998-824f-e8d425e414c3" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3"). InnerVolumeSpecName "kube-api-access-qj8nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:23:57.371591 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.371549    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9c2b47d0-5a18-4998-824f-e8d425e414c3" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:23:57.377257 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.377229    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2b47d0-5a18-4998-824f-e8d425e414c3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9c2b47d0-5a18-4998-824f-e8d425e414c3" (UID: "9c2b47d0-5a18-4998-824f-e8d425e414c3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:23:57.470135 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.470044    2567 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-bound-sa-token\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:57.470135 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.470075    2567 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-registry-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:57.470135 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.470084    2567 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-installation-pull-secrets\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:57.470135 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.470096    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj8nd\" (UniqueName: \"kubernetes.io/projected/9c2b47d0-5a18-4998-824f-e8d425e414c3-kube-api-access-qj8nd\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:57.470135 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.470106    2567 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c2b47d0-5a18-4998-824f-e8d425e414c3-image-registry-private-configuration\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:57.470135 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.470115    2567 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c2b47d0-5a18-4998-824f-e8d425e414c3-ca-trust-extracted\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:23:57.630674 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.630641    2567 generic.go:356] "Generic (PLEG): container finished" podID="9c2b47d0-5a18-4998-824f-e8d425e414c3" containerID="d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3" exitCode=0
Sep 16 21:23:57.630837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.630688    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f97687479-dkdrq"
Sep 16 21:23:57.630837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.630699    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f97687479-dkdrq" event={"ID":"9c2b47d0-5a18-4998-824f-e8d425e414c3","Type":"ContainerDied","Data":"d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3"}
Sep 16 21:23:57.630837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.630770    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f97687479-dkdrq" event={"ID":"9c2b47d0-5a18-4998-824f-e8d425e414c3","Type":"ContainerDied","Data":"7c887945c391cd069c21451dd5222f1e0214ac4ace3326b5a10ff9b40cd6c021"}
Sep 16 21:23:57.630837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.630793    2567 scope.go:117] "RemoveContainer" containerID="d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3"
Sep 16 21:23:57.639395 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.639379    2567 scope.go:117] "RemoveContainer" containerID="d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3"
Sep 16 21:23:57.639668 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:23:57.639647    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3\": container with ID starting with d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3 not found: ID does not exist" containerID="d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3"
Sep 16 21:23:57.639760 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.639674    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3"} err="failed to get container status \"d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3\": rpc error: code = NotFound desc = could not find container \"d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3\": container with ID starting with d024829f75eab0693666532afb38edb0b5f6a424e47dd7b239fe37eb6374d3e3 not found: ID does not exist"
Sep 16 21:23:57.652711 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.652671    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f97687479-dkdrq"]
Sep 16 21:23:57.659167 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.659136    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f97687479-dkdrq"]
Sep 16 21:23:57.942207 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:23:57.942176    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c2b47d0-5a18-4998-824f-e8d425e414c3" path="/var/lib/kubelet/pods/9c2b47d0-5a18-4998-824f-e8d425e414c3/volumes"
Sep 16 21:24:37.275496 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:37.275456    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:24:40.825122 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:40.825080    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:24:40.888519 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:40.888477    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:24:43.848386 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:43.848344    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:24:43.903911 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:43.903858    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:24:52.794199 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:52.794149    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:24:52.846771 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:52.846734    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:24:54.473728 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:54.473689    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:24:54.533089 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:54.533056    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:24:54.594970 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:24:54.594936    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:25:05.301995 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:25:05.301957    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:25:05.402058 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:25:05.402019    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:25:25.641451 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:25:25.641414    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:25:25.692253 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:25:25.692219    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:25:55.960821 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:25:55.960786    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:25:56.055534 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:25:56.055466    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:17.973799 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:17.973714    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:19.389257 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:19.389218    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:21.591463 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:21.591422    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:21.692376 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:21.692344    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:23.590368 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:23.590330    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:23.761494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:23.761457    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:23.875725 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:23.875642    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:23.967990 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:23.967949    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:26.216574 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:26.216535    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:26.427414 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:26.427379    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:28.797537 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:28.797497    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:28.857254 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:28.857216    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:33.811097 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:33.811046    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:36.460022 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:36.459976    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:36.560623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:36.560590    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:41.988445 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:41.988406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:42.111151 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:42.111111    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:44.014825 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:44.014784    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:47.411757 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:47.411722    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:47.504465 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:47.504435    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:26:53.767806 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:53.767775    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:26:53.774338 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:53.774312    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:26:53.783730 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:53.783708    2567 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 21:26:59.817690 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:26:59.817654    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:01.026736 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.026700    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"]
Sep 16 21:27:01.027178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.027021    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c2b47d0-5a18-4998-824f-e8d425e414c3" containerName="registry"
Sep 16 21:27:01.027178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.027034    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c2b47d0-5a18-4998-824f-e8d425e414c3" containerName="registry"
Sep 16 21:27:01.027178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.027054    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90ad643-4367-4547-80b1-d72f7e9f3347" containerName="console"
Sep 16 21:27:01.027178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.027059    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90ad643-4367-4547-80b1-d72f7e9f3347" containerName="console"
Sep 16 21:27:01.027178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.027117    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90ad643-4367-4547-80b1-d72f7e9f3347" containerName="console"
Sep 16 21:27:01.027178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.027127    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c2b47d0-5a18-4998-824f-e8d425e414c3" containerName="registry"
Sep 16 21:27:01.029598 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.029576    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.031980 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.031957    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 21:27:01.032146 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.032123    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 21:27:01.032476 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.032456    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 21:27:01.032556 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.032477    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 21:27:01.032556 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.032509    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lk5q9\""
Sep 16 21:27:01.055687 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.055649    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"]
Sep 16 21:27:01.160634 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.160596    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5f4ab67b-5713-4d86-b4ab-b9081e9da416-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.160813 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.160642    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f4ab67b-5713-4d86-b4ab-b9081e9da416-tmp\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.160813 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.160673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5f4ab67b-5713-4d86-b4ab-b9081e9da416-manifests\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.160813 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.160706    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hctj\" (UniqueName: \"kubernetes.io/projected/5f4ab67b-5713-4d86-b4ab-b9081e9da416-kube-api-access-6hctj\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.160813 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.160727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f4ab67b-5713-4d86-b4ab-b9081e9da416-webhook-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.172834 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.172799    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:01.261687 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.261645    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f4ab67b-5713-4d86-b4ab-b9081e9da416-tmp\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.261687 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.261686    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5f4ab67b-5713-4d86-b4ab-b9081e9da416-manifests\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.261985 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.261714    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hctj\" (UniqueName: \"kubernetes.io/projected/5f4ab67b-5713-4d86-b4ab-b9081e9da416-kube-api-access-6hctj\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.261985 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.261774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f4ab67b-5713-4d86-b4ab-b9081e9da416-webhook-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.261985 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.261927    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5f4ab67b-5713-4d86-b4ab-b9081e9da416-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.262273 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.262247    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5f4ab67b-5713-4d86-b4ab-b9081e9da416-manifests\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.264063 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.264030    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f4ab67b-5713-4d86-b4ab-b9081e9da416-tmp\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.264260 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.264242    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5f4ab67b-5713-4d86-b4ab-b9081e9da416-webhook-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.264383 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.264365    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5f4ab67b-5713-4d86-b4ab-b9081e9da416-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.270448 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.270425    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hctj\" (UniqueName: \"kubernetes.io/projected/5f4ab67b-5713-4d86-b4ab-b9081e9da416-kube-api-access-6hctj\") pod \"opendatahub-operator-controller-manager-df9d657c6-dkns4\" (UID: \"5f4ab67b-5713-4d86-b4ab-b9081e9da416\") " pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.272718 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.272694    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:01.340672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.340588    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:01.362355 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.362323    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:01.426445 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.426415    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:01.528815 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.528777    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:01.609060 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.608957    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:01.622530 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.622499    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"]
Sep 16 21:27:01.626116 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:27:01.626084    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4ab67b_5713_4d86_b4ab_b9081e9da416.slice/crio-0582ef51cba6de2504b928511565869613e449bcf7d5658e2ac29f1d48e099d7 WatchSource:0}: Error finding container 0582ef51cba6de2504b928511565869613e449bcf7d5658e2ac29f1d48e099d7: Status 404 returned error can't find the container with id 0582ef51cba6de2504b928511565869613e449bcf7d5658e2ac29f1d48e099d7
Sep 16 21:27:01.628465 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:01.628444    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:27:02.365216 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:02.365174    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4" event={"ID":"5f4ab67b-5713-4d86-b4ab-b9081e9da416","Type":"ContainerStarted","Data":"0582ef51cba6de2504b928511565869613e449bcf7d5658e2ac29f1d48e099d7"}
Sep 16 21:27:05.382276 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:05.382242    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4" event={"ID":"5f4ab67b-5713-4d86-b4ab-b9081e9da416","Type":"ContainerStarted","Data":"a6e5894e2f91aee99feafe33c2fceb96c9d27c99de27c76f1428a501c6cba80f"}
Sep 16 21:27:06.391119 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:06.391081    2567 generic.go:356] "Generic (PLEG): container finished" podID="5f4ab67b-5713-4d86-b4ab-b9081e9da416" containerID="a6e5894e2f91aee99feafe33c2fceb96c9d27c99de27c76f1428a501c6cba80f" exitCode=0
Sep 16 21:27:06.391597 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:06.391165    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4" event={"ID":"5f4ab67b-5713-4d86-b4ab-b9081e9da416","Type":"ContainerDied","Data":"a6e5894e2f91aee99feafe33c2fceb96c9d27c99de27c76f1428a501c6cba80f"}
Sep 16 21:27:07.397222 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:07.397178    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4" event={"ID":"5f4ab67b-5713-4d86-b4ab-b9081e9da416","Type":"ContainerStarted","Data":"7c5a8da75a44e011f2f05193f108e7d34e5250067761cb691a11522f01337c59"}
Sep 16 21:27:07.397598 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:07.397294    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:07.421270 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:07.421223    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4" podStartSLOduration=2.035993216 podStartE2EDuration="6.421206597s" totalImagesPullingTime="3.365440502s" totalInitContainerRuntime="1.019772879s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:07.418889459 +0000 UTC m=+314.067301448" watchObservedRunningTime="2026-09-16 21:27:07.421206597 +0000 UTC m=+314.069618646"
Sep 16 21:27:10.213326 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:10.213286    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:10.314602 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:10.314565    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:13.231521 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:13.231480    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:13.332527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:13.332494    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:18.403320 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:18.403284    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-df9d657c6-dkns4"
Sep 16 21:27:20.044977 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:20.044936    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:20.185440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:20.185404    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:20.350966 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:20.350859    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:20.441444 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:20.441408    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:20.925418 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:20.925379    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:21.636470 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:21.636431    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:21.707371 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:21.707332    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:22.537122 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:22.537081    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:26.343854 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:26.343814    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:26.425850 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:26.425797    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:26.951955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:26.951912    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:27.119863 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:27.119826    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:27.216056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:27.215965    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:31.327894 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:31.327855    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:31.427534 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:31.427456    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:31.527796 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:31.527762    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:31.960156 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:31.960123    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:32.055704 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:32.055669    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:38.660017 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.659980    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"]
Sep 16 21:27:38.668578 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.668549    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.673078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.673049    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 16 21:27:38.673213 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.673082    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 16 21:27:38.673213 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.673106    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qlc8g\""
Sep 16 21:27:38.673213 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.673115    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 16 21:27:38.673213 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.673112    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 16 21:27:38.673213 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.673160    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 16 21:27:38.682291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.682264    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"]
Sep 16 21:27:38.742714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.742679    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-metrics-cert\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.743001 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.742779    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxrc\" (UniqueName: \"kubernetes.io/projected/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-kube-api-access-ksxrc\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.743001 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.742812    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-cert\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.743001 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.742865    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-manager-config\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.843819 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.843786    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxrc\" (UniqueName: \"kubernetes.io/projected/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-kube-api-access-ksxrc\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.843819 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.843818    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-cert\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.844092 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.843844    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-manager-config\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.844092 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.843870    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-metrics-cert\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.844518 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.844493    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-manager-config\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.846498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.846474    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-cert\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.846588 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.846482    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-metrics-cert\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.852139 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.852111    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxrc\" (UniqueName: \"kubernetes.io/projected/1ca96afd-9e20-4557-8b0a-a6f1e2d14569-kube-api-access-ksxrc\") pod \"lws-controller-manager-5dfdd9df77-8nh9g\" (UID: \"1ca96afd-9e20-4557-8b0a-a6f1e2d14569\") " pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:38.977974 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:38.977876    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:39.258616 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:39.258545    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"]
Sep 16 21:27:39.262247 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:27:39.262217    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca96afd_9e20_4557_8b0a_a6f1e2d14569.slice/crio-db0dbda86173f1fcec3d3777dd382fcb9e74289b4ac9e2b5e7dd6bbd55d7569b WatchSource:0}: Error finding container db0dbda86173f1fcec3d3777dd382fcb9e74289b4ac9e2b5e7dd6bbd55d7569b: Status 404 returned error can't find the container with id db0dbda86173f1fcec3d3777dd382fcb9e74289b4ac9e2b5e7dd6bbd55d7569b
Sep 16 21:27:39.543817 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:39.543735    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g" event={"ID":"1ca96afd-9e20-4557-8b0a-a6f1e2d14569","Type":"ContainerStarted","Data":"db0dbda86173f1fcec3d3777dd382fcb9e74289b4ac9e2b5e7dd6bbd55d7569b"}
Sep 16 21:27:41.747203 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:41.747123    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:41.900358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:41.900308    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:42.293050 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:42.292961    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:42.466388 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:42.466352    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:42.559783 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:42.559694    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g" event={"ID":"1ca96afd-9e20-4557-8b0a-a6f1e2d14569","Type":"ContainerStarted","Data":"013080c1941d906a9e78f7171979d373254b4e415f201cdad01c3c5fd4b9eb4e"}
Sep 16 21:27:42.559962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:42.559832    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:27:42.581366 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:42.581314    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g" podStartSLOduration=1.873049016 podStartE2EDuration="4.58129871s" totalImagesPullingTime="2.708249694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:27:42.580153329 +0000 UTC m=+349.228565319" watchObservedRunningTime="2026-09-16 21:27:42.58129871 +0000 UTC m=+349.229710698"
Sep 16 21:27:43.079250 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:43.079216    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:43.223627 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:43.223583    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:43.310042 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:43.310002    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:27:53.567318 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:27:53.567287    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5dfdd9df77-8nh9g"
Sep 16 21:28:02.392555 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:02.392516    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:11.589253 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.589189    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ddcb64c95-dvrhh"]
Sep 16 21:28:11.598639 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.598609    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.601697 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.601669    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 21:28:11.602111 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.601713    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 21:28:11.602228 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.601730    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 21:28:11.603818 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.603525    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 21:28:11.603818 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.603658    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vpfpz\""
Sep 16 21:28:11.603818 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.603754    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 21:28:11.604527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.604446    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ddcb64c95-dvrhh"]
Sep 16 21:28:11.608347 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.608324    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 21:28:11.653296 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.653261    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:11.718563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.718529    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-service-ca\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.718735 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.718574    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13b099cb-ff08-465b-8be2-00b74b0977b1-console-oauth-config\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.718735 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.718633    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13b099cb-ff08-465b-8be2-00b74b0977b1-console-serving-cert\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.718735 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.718648    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmpf2\" (UniqueName: \"kubernetes.io/projected/13b099cb-ff08-465b-8be2-00b74b0977b1-kube-api-access-pmpf2\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.718735 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.718701    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-trusted-ca-bundle\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.718735 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.718721    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-console-config\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.718946 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.718737    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-oauth-serving-cert\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.734729 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.734697    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:11.797126 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.797092    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:11.819301 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.819268    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-service-ca\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.819462 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.819310    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13b099cb-ff08-465b-8be2-00b74b0977b1-console-oauth-config\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.819462 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.819383    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13b099cb-ff08-465b-8be2-00b74b0977b1-console-serving-cert\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.819462 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.819411    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmpf2\" (UniqueName: \"kubernetes.io/projected/13b099cb-ff08-465b-8be2-00b74b0977b1-kube-api-access-pmpf2\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.819608 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.819477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-trusted-ca-bundle\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.819689 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.819664    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-console-config\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.819738 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.819707    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-oauth-serving-cert\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.820280 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.820253    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-service-ca\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.820476 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.820450    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-oauth-serving-cert\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.820550 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.820483    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-console-config\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.820614 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.820589    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13b099cb-ff08-465b-8be2-00b74b0977b1-trusted-ca-bundle\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.821983 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.821964    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/13b099cb-ff08-465b-8be2-00b74b0977b1-console-oauth-config\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.822078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.822067    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/13b099cb-ff08-465b-8be2-00b74b0977b1-console-serving-cert\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.827538 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.827520    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmpf2\" (UniqueName: \"kubernetes.io/projected/13b099cb-ff08-465b-8be2-00b74b0977b1-kube-api-access-pmpf2\") pod \"console-5ddcb64c95-dvrhh\" (UID: \"13b099cb-ff08-465b-8be2-00b74b0977b1\") " pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:11.910849 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:11.910749    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:12.188645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:12.188567    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ddcb64c95-dvrhh"]
Sep 16 21:28:12.191873 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:28:12.191837    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b099cb_ff08_465b_8be2_00b74b0977b1.slice/crio-8e6ea026a1c43d79399de526774bb1686a9166bd23e9cf7185c991445e55bbb1 WatchSource:0}: Error finding container 8e6ea026a1c43d79399de526774bb1686a9166bd23e9cf7185c991445e55bbb1: Status 404 returned error can't find the container with id 8e6ea026a1c43d79399de526774bb1686a9166bd23e9cf7185c991445e55bbb1
Sep 16 21:28:12.387776 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:12.387739    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:12.441094 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:12.441003    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:12.694040 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:12.693939    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddcb64c95-dvrhh" event={"ID":"13b099cb-ff08-465b-8be2-00b74b0977b1","Type":"ContainerStarted","Data":"adb71cb547d298ff0207f924c14f9aad514eff79bb699300d40dda54a7f292b3"}
Sep 16 21:28:12.694040 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:12.694004    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ddcb64c95-dvrhh" event={"ID":"13b099cb-ff08-465b-8be2-00b74b0977b1","Type":"ContainerStarted","Data":"8e6ea026a1c43d79399de526774bb1686a9166bd23e9cf7185c991445e55bbb1"}
Sep 16 21:28:12.712399 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:12.712347    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ddcb64c95-dvrhh" podStartSLOduration=1.7123294169999999 podStartE2EDuration="1.712329417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:28:12.711611381 +0000 UTC m=+379.360023369" watchObservedRunningTime="2026-09-16 21:28:12.712329417 +0000 UTC m=+379.360741404"
Sep 16 21:28:21.911357 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:21.911316    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:21.911761 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:21.911381    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:21.916522 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:21.916493    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:22.740056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:22.740023    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ddcb64c95-dvrhh"
Sep 16 21:28:22.888483 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:22.888447    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:22.951831 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:22.951786    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:23.007212 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:23.007186    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:32.145478 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:32.145429    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:32.244864 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:32.244824    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:40.202065 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:40.202027    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:56.382629 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:56.382597    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:57.177237 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:57.177204    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:28:57.990604 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:28:57.990575    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:01.137741 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:01.137708    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:01.242924 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:01.242879    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:01.475691 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:01.475609    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:01.589757 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:01.589725    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:01.691266 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:01.691229    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:02.629359 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:02.629318    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:03.512026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.511993    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bdtzc"]
Sep 16 21:29:03.514976 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.514961    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc"
Sep 16 21:29:03.519047 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.519026    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l8fc6\""
Sep 16 21:29:03.519217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.519194    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 21:29:03.519689 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.519675    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 21:29:03.541284 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.541251    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bdtzc"]
Sep 16 21:29:03.552624 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.552594    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99g5\" (UniqueName: \"kubernetes.io/projected/6b6404bd-dfa5-4f11-a268-f0b39315000a-kube-api-access-j99g5\") pod \"authorino-operator-76b478bd78-bdtzc\" (UID: \"6b6404bd-dfa5-4f11-a268-f0b39315000a\") " pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc"
Sep 16 21:29:03.553438 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.553417    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:03.653575 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.653530    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99g5\" (UniqueName: \"kubernetes.io/projected/6b6404bd-dfa5-4f11-a268-f0b39315000a-kube-api-access-j99g5\") pod \"authorino-operator-76b478bd78-bdtzc\" (UID: \"6b6404bd-dfa5-4f11-a268-f0b39315000a\") " pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc"
Sep 16 21:29:03.663425 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.663398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99g5\" (UniqueName: \"kubernetes.io/projected/6b6404bd-dfa5-4f11-a268-f0b39315000a-kube-api-access-j99g5\") pod \"authorino-operator-76b478bd78-bdtzc\" (UID: \"6b6404bd-dfa5-4f11-a268-f0b39315000a\") " pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc"
Sep 16 21:29:03.825012 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:03.824928    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc"
Sep 16 21:29:04.104569 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:04.104487    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-bdtzc"]
Sep 16 21:29:04.107369 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:29:04.107346    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6404bd_dfa5_4f11_a268_f0b39315000a.slice/crio-9de3f3500076d68aff059eb504a49aab1b9af70fddb8e82c012410db78105613 WatchSource:0}: Error finding container 9de3f3500076d68aff059eb504a49aab1b9af70fddb8e82c012410db78105613: Status 404 returned error can't find the container with id 9de3f3500076d68aff059eb504a49aab1b9af70fddb8e82c012410db78105613
Sep 16 21:29:04.254242 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:04.254194    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:04.464304 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:04.464219    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:04.925429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:04.925396    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc" event={"ID":"6b6404bd-dfa5-4f11-a268-f0b39315000a","Type":"ContainerStarted","Data":"9de3f3500076d68aff059eb504a49aab1b9af70fddb8e82c012410db78105613"}
Sep 16 21:29:06.405226 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:06.405187    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:07.855179 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:07.855139    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:07.941795 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:07.941759    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc" event={"ID":"6b6404bd-dfa5-4f11-a268-f0b39315000a","Type":"ContainerStarted","Data":"ebfac6f39e239fc10aad72684e546ac8281fecbeeda71549aad8ce3078313457"}
Sep 16 21:29:07.941967 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:07.941813    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc"
Sep 16 21:29:07.962384 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:07.962337    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc" podStartSLOduration=1.973727067 podStartE2EDuration="4.96232339s" totalImagesPullingTime="2.988596323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:07.960434463 +0000 UTC m=+434.608846452" watchObservedRunningTime="2026-09-16 21:29:07.96232339 +0000 UTC m=+434.610735377"
Sep 16 21:29:09.663347 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:09.663316    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:09.749781 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:09.749747    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:11.595155 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:11.595123    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:11.765704 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:11.765666    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:11.998019 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:11.997989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:12.056293 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:12.056263    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:12.389693 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:12.389601    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:12.445150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:12.445119    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:18.949060 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:18.948979    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-bdtzc"
Sep 16 21:29:20.492486 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:20.492447    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:20.736972 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:20.736937    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:22.877247 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:22.877209    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:22.975284 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:22.975241    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:27.702225 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:27.702182    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:27.792417 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:27.792383    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:32.826770 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:32.826735    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:32.935788 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:32.935756    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:33.235703 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:33.235610    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:33.335437 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:33.335404    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:33.445264 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:33.445228    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:33.586798 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:33.586765    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:42.212132 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:42.212097    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:42.312221 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:42.312189    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:44.670585 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:44.670546    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:44.770692 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:44.770656    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:48.574806 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:48.574769    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:49.227747 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:49.227703    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:49.420325 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:49.420291    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:49.558918 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:49.558860    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:50.854943 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.854893    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mww5z"]
Sep 16 21:29:50.855947 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.855917    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:50.857538 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.857523    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:29:50.859869 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.859848    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 21:29:50.860136 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.860121    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hkhcz\""
Sep 16 21:29:50.867280 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.867258    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mww5z"]
Sep 16 21:29:50.934349 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.934312    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mww5z"]
Sep 16 21:29:50.939044 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.939021    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a85d052f-aec0-46ac-89a7-5e1456a8131c-config-file\") pod \"limitador-limitador-8588749977-mww5z\" (UID: \"a85d052f-aec0-46ac-89a7-5e1456a8131c\") " pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:29:50.939157 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.939055    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b29nk\" (UniqueName: \"kubernetes.io/projected/a85d052f-aec0-46ac-89a7-5e1456a8131c-kube-api-access-b29nk\") pod \"limitador-limitador-8588749977-mww5z\" (UID: \"a85d052f-aec0-46ac-89a7-5e1456a8131c\") " pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:29:50.942381 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:50.942350    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:51.002221 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.002190    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:51.040514 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.040482    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a85d052f-aec0-46ac-89a7-5e1456a8131c-config-file\") pod \"limitador-limitador-8588749977-mww5z\" (UID: \"a85d052f-aec0-46ac-89a7-5e1456a8131c\") " pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:29:51.040514 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.040522    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b29nk\" (UniqueName: \"kubernetes.io/projected/a85d052f-aec0-46ac-89a7-5e1456a8131c-kube-api-access-b29nk\") pod \"limitador-limitador-8588749977-mww5z\" (UID: \"a85d052f-aec0-46ac-89a7-5e1456a8131c\") " pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:29:51.041228 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.041199    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a85d052f-aec0-46ac-89a7-5e1456a8131c-config-file\") pod \"limitador-limitador-8588749977-mww5z\" (UID: \"a85d052f-aec0-46ac-89a7-5e1456a8131c\") " pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:29:51.048544 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.048517    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b29nk\" (UniqueName: \"kubernetes.io/projected/a85d052f-aec0-46ac-89a7-5e1456a8131c-kube-api-access-b29nk\") pod \"limitador-limitador-8588749977-mww5z\" (UID: \"a85d052f-aec0-46ac-89a7-5e1456a8131c\") " pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:29:51.169021 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.168932    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:29:51.438525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.438395    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mww5z"]
Sep 16 21:29:51.441379 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:29:51.441350    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85d052f_aec0_46ac_89a7_5e1456a8131c.slice/crio-f9c07ad9ca9e267fb53390937e097c11575c3e750a9ee11ae8ccc8015815931e WatchSource:0}: Error finding container f9c07ad9ca9e267fb53390937e097c11575c3e750a9ee11ae8ccc8015815931e: Status 404 returned error can't find the container with id f9c07ad9ca9e267fb53390937e097c11575c3e750a9ee11ae8ccc8015815931e
Sep 16 21:29:51.717865 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.717753    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:51.818574 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.818539    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:29:51.874780 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.874743    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-qt792"]
Sep 16 21:29:51.877885 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.877865    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qt792"
Sep 16 21:29:51.880105 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.880090    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6m8j5\""
Sep 16 21:29:51.885026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.885002    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qt792"]
Sep 16 21:29:51.948340 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:51.948312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bd8\" (UniqueName: \"kubernetes.io/projected/a0946411-948d-4d20-adb5-3cc629fd7dd5-kube-api-access-t2bd8\") pod \"authorino-59c974bd48-qt792\" (UID: \"a0946411-948d-4d20-adb5-3cc629fd7dd5\") " pod="kuadrant-system/authorino-59c974bd48-qt792"
Sep 16 21:29:52.049703 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:52.049665    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bd8\" (UniqueName: \"kubernetes.io/projected/a0946411-948d-4d20-adb5-3cc629fd7dd5-kube-api-access-t2bd8\") pod \"authorino-59c974bd48-qt792\" (UID: \"a0946411-948d-4d20-adb5-3cc629fd7dd5\") " pod="kuadrant-system/authorino-59c974bd48-qt792"
Sep 16 21:29:52.058387 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:52.058352    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bd8\" (UniqueName: \"kubernetes.io/projected/a0946411-948d-4d20-adb5-3cc629fd7dd5-kube-api-access-t2bd8\") pod \"authorino-59c974bd48-qt792\" (UID: \"a0946411-948d-4d20-adb5-3cc629fd7dd5\") " pod="kuadrant-system/authorino-59c974bd48-qt792"
Sep 16 21:29:52.134230 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:52.134194    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mww5z" event={"ID":"a85d052f-aec0-46ac-89a7-5e1456a8131c","Type":"ContainerStarted","Data":"f9c07ad9ca9e267fb53390937e097c11575c3e750a9ee11ae8ccc8015815931e"}
Sep 16 21:29:52.187842 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:52.187812    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qt792"
Sep 16 21:29:52.455874 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:52.455847    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qt792"]
Sep 16 21:29:52.457990 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:29:52.457952    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0946411_948d_4d20_adb5_3cc629fd7dd5.slice/crio-f12ce017c7e62312dce9c690de734f4ada5a5849df301e0ca6c73c6e38ee4b54 WatchSource:0}: Error finding container f12ce017c7e62312dce9c690de734f4ada5a5849df301e0ca6c73c6e38ee4b54: Status 404 returned error can't find the container with id f12ce017c7e62312dce9c690de734f4ada5a5849df301e0ca6c73c6e38ee4b54
Sep 16 21:29:53.140735 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:53.140699    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mww5z" event={"ID":"a85d052f-aec0-46ac-89a7-5e1456a8131c","Type":"ContainerStarted","Data":"d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594"}
Sep 16 21:29:53.141207 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:53.140759    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:29:53.141970 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:53.141952    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qt792" event={"ID":"a0946411-948d-4d20-adb5-3cc629fd7dd5","Type":"ContainerStarted","Data":"f12ce017c7e62312dce9c690de734f4ada5a5849df301e0ca6c73c6e38ee4b54"}
Sep 16 21:29:53.157748 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:53.157706    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-mww5z" podStartSLOduration=1.628155223 podStartE2EDuration="3.157693048s" totalImagesPullingTime="1.529537825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:53.157074211 +0000 UTC m=+479.805486199" watchObservedRunningTime="2026-09-16 21:29:53.157693048 +0000 UTC m=+479.806105035"
Sep 16 21:29:55.153230 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:55.153143    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qt792" event={"ID":"a0946411-948d-4d20-adb5-3cc629fd7dd5","Type":"ContainerStarted","Data":"38d24e8ef013486cd8553d8f3add4eaa78aee44e5f0082c09644370b494ca5ca"}
Sep 16 21:29:55.169201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:55.169148    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-qt792" podStartSLOduration=1.840194693 podStartE2EDuration="4.169132296s" totalImagesPullingTime="2.328937603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:29:55.168371318 +0000 UTC m=+481.816783307" watchObservedRunningTime="2026-09-16 21:29:55.169132296 +0000 UTC m=+481.817544283"
Sep 16 21:29:59.399563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:29:59.399525    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:01.197740 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:01.197702    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:01.299630 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:01.299597    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:04.147579 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:04.147547    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:30:04.366989 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:04.366944    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mww5z"]
Sep 16 21:30:04.367229 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:04.367206    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-mww5z" podUID="a85d052f-aec0-46ac-89a7-5e1456a8131c" containerName="limitador" containerID="cri-o://d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594" gracePeriod=30
Sep 16 21:30:04.916128 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:04.916102    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:30:04.961364 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:04.961271    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b29nk\" (UniqueName: \"kubernetes.io/projected/a85d052f-aec0-46ac-89a7-5e1456a8131c-kube-api-access-b29nk\") pod \"a85d052f-aec0-46ac-89a7-5e1456a8131c\" (UID: \"a85d052f-aec0-46ac-89a7-5e1456a8131c\") "
Sep 16 21:30:04.961364 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:04.961315    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a85d052f-aec0-46ac-89a7-5e1456a8131c-config-file\") pod \"a85d052f-aec0-46ac-89a7-5e1456a8131c\" (UID: \"a85d052f-aec0-46ac-89a7-5e1456a8131c\") "
Sep 16 21:30:04.961686 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:04.961656    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85d052f-aec0-46ac-89a7-5e1456a8131c-config-file" (OuterVolumeSpecName: "config-file") pod "a85d052f-aec0-46ac-89a7-5e1456a8131c" (UID: "a85d052f-aec0-46ac-89a7-5e1456a8131c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:30:04.963493 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:04.963471    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85d052f-aec0-46ac-89a7-5e1456a8131c-kube-api-access-b29nk" (OuterVolumeSpecName: "kube-api-access-b29nk") pod "a85d052f-aec0-46ac-89a7-5e1456a8131c" (UID: "a85d052f-aec0-46ac-89a7-5e1456a8131c"). InnerVolumeSpecName "kube-api-access-b29nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:30:05.062288 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.062245    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b29nk\" (UniqueName: \"kubernetes.io/projected/a85d052f-aec0-46ac-89a7-5e1456a8131c-kube-api-access-b29nk\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:30:05.062288 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.062281    2567 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a85d052f-aec0-46ac-89a7-5e1456a8131c-config-file\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:30:05.201057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.201021    2567 generic.go:356] "Generic (PLEG): container finished" podID="a85d052f-aec0-46ac-89a7-5e1456a8131c" containerID="d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594" exitCode=0
Sep 16 21:30:05.201525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.201065    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mww5z"
Sep 16 21:30:05.201525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.201115    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mww5z" event={"ID":"a85d052f-aec0-46ac-89a7-5e1456a8131c","Type":"ContainerDied","Data":"d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594"}
Sep 16 21:30:05.201525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.201177    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mww5z" event={"ID":"a85d052f-aec0-46ac-89a7-5e1456a8131c","Type":"ContainerDied","Data":"f9c07ad9ca9e267fb53390937e097c11575c3e750a9ee11ae8ccc8015815931e"}
Sep 16 21:30:05.201525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.201214    2567 scope.go:117] "RemoveContainer" containerID="d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594"
Sep 16 21:30:05.210854 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.210836    2567 scope.go:117] "RemoveContainer" containerID="d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594"
Sep 16 21:30:05.211177 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:30:05.211160    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594\": container with ID starting with d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594 not found: ID does not exist" containerID="d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594"
Sep 16 21:30:05.211232 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.211186    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594"} err="failed to get container status \"d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594\": rpc error: code = NotFound desc = could not find container \"d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594\": container with ID starting with d5fb96ecb90a30a74ddded3db549a23c1294718e7502803f52b71758fbc66594 not found: ID does not exist"
Sep 16 21:30:05.222496 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.222447    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mww5z"]
Sep 16 21:30:05.226090 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.226068    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mww5z"]
Sep 16 21:30:05.941330 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:05.941297    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85d052f-aec0-46ac-89a7-5e1456a8131c" path="/var/lib/kubelet/pods/a85d052f-aec0-46ac-89a7-5e1456a8131c/volumes"
Sep 16 21:30:06.803971 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:06.803928    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:06.904272 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:06.904216    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:07.278978 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.278944    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"]
Sep 16 21:30:07.279470 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.279450    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85d052f-aec0-46ac-89a7-5e1456a8131c" containerName="limitador"
Sep 16 21:30:07.279549 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.279473    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85d052f-aec0-46ac-89a7-5e1456a8131c" containerName="limitador"
Sep 16 21:30:07.279605 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.279577    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85d052f-aec0-46ac-89a7-5e1456a8131c" containerName="limitador"
Sep 16 21:30:07.284431 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.284406    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:07.287183 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.286996    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 21:30:07.287183 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.287036    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-fzmc4\""
Sep 16 21:30:07.287183 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.287048    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 21:30:07.287963 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.287942    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 21:30:07.292330 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.292178    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"]
Sep 16 21:30:07.327365 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.327327    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:07.382311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.382275    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjn9\" (UniqueName: \"kubernetes.io/projected/117a9d10-99ce-473f-8b93-2b022da5f498-kube-api-access-wvjn9\") pod \"postgres-69fb4594cf-nnj72\" (UID: \"117a9d10-99ce-473f-8b93-2b022da5f498\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:07.382528 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.382327    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/117a9d10-99ce-473f-8b93-2b022da5f498-data\") pod \"postgres-69fb4594cf-nnj72\" (UID: \"117a9d10-99ce-473f-8b93-2b022da5f498\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:07.483713 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.483670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjn9\" (UniqueName: \"kubernetes.io/projected/117a9d10-99ce-473f-8b93-2b022da5f498-kube-api-access-wvjn9\") pod \"postgres-69fb4594cf-nnj72\" (UID: \"117a9d10-99ce-473f-8b93-2b022da5f498\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:07.483888 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.483725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/117a9d10-99ce-473f-8b93-2b022da5f498-data\") pod \"postgres-69fb4594cf-nnj72\" (UID: \"117a9d10-99ce-473f-8b93-2b022da5f498\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:07.484156 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.484136    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/117a9d10-99ce-473f-8b93-2b022da5f498-data\") pod \"postgres-69fb4594cf-nnj72\" (UID: \"117a9d10-99ce-473f-8b93-2b022da5f498\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:07.491877 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.491851    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjn9\" (UniqueName: \"kubernetes.io/projected/117a9d10-99ce-473f-8b93-2b022da5f498-kube-api-access-wvjn9\") pod \"postgres-69fb4594cf-nnj72\" (UID: \"117a9d10-99ce-473f-8b93-2b022da5f498\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:07.596778 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.596680    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:07.866201 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:07.866117    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"]
Sep 16 21:30:07.869636 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:30:07.869593    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117a9d10_99ce_473f_8b93_2b022da5f498.slice/crio-7f8d5d27d35daf50d1d535fb8cfaf7e8f24c1e006c2d57c292a840491abd0f8c WatchSource:0}: Error finding container 7f8d5d27d35daf50d1d535fb8cfaf7e8f24c1e006c2d57c292a840491abd0f8c: Status 404 returned error can't find the container with id 7f8d5d27d35daf50d1d535fb8cfaf7e8f24c1e006c2d57c292a840491abd0f8c
Sep 16 21:30:08.216768 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:08.216683    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72" event={"ID":"117a9d10-99ce-473f-8b93-2b022da5f498","Type":"ContainerStarted","Data":"7f8d5d27d35daf50d1d535fb8cfaf7e8f24c1e006c2d57c292a840491abd0f8c"}
Sep 16 21:30:14.252926 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:14.252869    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72" event={"ID":"117a9d10-99ce-473f-8b93-2b022da5f498","Type":"ContainerStarted","Data":"46688b9033358399ccfb0c0d48915f3e6ab9690f51d35f3bbe39534314b16924"}
Sep 16 21:30:14.253353 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:14.252955    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:14.270849 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:14.270776    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72" podStartSLOduration=1.680007796 podStartE2EDuration="7.270752899s" totalImagesPullingTime="5.590745103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:30:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:30:14.268505178 +0000 UTC m=+500.916917164" watchObservedRunningTime="2026-09-16 21:30:14.270752899 +0000 UTC m=+500.919164887"
Sep 16 21:30:20.294037 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:20.294007    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-nnj72"
Sep 16 21:30:20.600864 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:20.600772    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:20.887864 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:20.887788    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:20.949179 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:20.949140    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jtc7s"]
Sep 16 21:30:20.952724 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:20.952698    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:20.955199 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:20.955178    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 21:30:20.961428 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:20.961407    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jtc7s"]
Sep 16 21:30:21.063241 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.063202    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jtc7s"]
Sep 16 21:30:21.063491 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:30:21.063469    2567 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4bp46 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-jtc7s" podUID="7d7234df-14a4-43df-83e4-36d6dc71c067"
Sep 16 21:30:21.109008 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.108969    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bp46\" (UniqueName: \"kubernetes.io/projected/7d7234df-14a4-43df-83e4-36d6dc71c067-kube-api-access-4bp46\") pod \"authorino-6d4dfc649f-jtc7s\" (UID: \"7d7234df-14a4-43df-83e4-36d6dc71c067\") " pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:21.109165 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.109064    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d7234df-14a4-43df-83e4-36d6dc71c067-tls-cert\") pod \"authorino-6d4dfc649f-jtc7s\" (UID: \"7d7234df-14a4-43df-83e4-36d6dc71c067\") " pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:21.210576 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.210481    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bp46\" (UniqueName: \"kubernetes.io/projected/7d7234df-14a4-43df-83e4-36d6dc71c067-kube-api-access-4bp46\") pod \"authorino-6d4dfc649f-jtc7s\" (UID: \"7d7234df-14a4-43df-83e4-36d6dc71c067\") " pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:21.210754 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.210578    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d7234df-14a4-43df-83e4-36d6dc71c067-tls-cert\") pod \"authorino-6d4dfc649f-jtc7s\" (UID: \"7d7234df-14a4-43df-83e4-36d6dc71c067\") " pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:21.213083 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.213058    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d7234df-14a4-43df-83e4-36d6dc71c067-tls-cert\") pod \"authorino-6d4dfc649f-jtc7s\" (UID: \"7d7234df-14a4-43df-83e4-36d6dc71c067\") " pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:21.218161 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.218136    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bp46\" (UniqueName: \"kubernetes.io/projected/7d7234df-14a4-43df-83e4-36d6dc71c067-kube-api-access-4bp46\") pod \"authorino-6d4dfc649f-jtc7s\" (UID: \"7d7234df-14a4-43df-83e4-36d6dc71c067\") " pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:21.290725 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.290697    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:21.297920 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.297879    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:21.412691 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.412643    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d7234df-14a4-43df-83e4-36d6dc71c067-tls-cert\") pod \"7d7234df-14a4-43df-83e4-36d6dc71c067\" (UID: \"7d7234df-14a4-43df-83e4-36d6dc71c067\") "
Sep 16 21:30:21.412889 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.412761    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bp46\" (UniqueName: \"kubernetes.io/projected/7d7234df-14a4-43df-83e4-36d6dc71c067-kube-api-access-4bp46\") pod \"7d7234df-14a4-43df-83e4-36d6dc71c067\" (UID: \"7d7234df-14a4-43df-83e4-36d6dc71c067\") "
Sep 16 21:30:21.414947 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.414916    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d7234df-14a4-43df-83e4-36d6dc71c067-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "7d7234df-14a4-43df-83e4-36d6dc71c067" (UID: "7d7234df-14a4-43df-83e4-36d6dc71c067"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:30:21.415068 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.414998    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d7234df-14a4-43df-83e4-36d6dc71c067-kube-api-access-4bp46" (OuterVolumeSpecName: "kube-api-access-4bp46") pod "7d7234df-14a4-43df-83e4-36d6dc71c067" (UID: "7d7234df-14a4-43df-83e4-36d6dc71c067"). InnerVolumeSpecName "kube-api-access-4bp46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:30:21.513698 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.513665    2567 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7d7234df-14a4-43df-83e4-36d6dc71c067-tls-cert\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:30:21.513698 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:21.513696    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bp46\" (UniqueName: \"kubernetes.io/projected/7d7234df-14a4-43df-83e4-36d6dc71c067-kube-api-access-4bp46\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:30:22.295274 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:22.295243    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-jtc7s"
Sep 16 21:30:22.321039 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:22.321000    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jtc7s"]
Sep 16 21:30:22.323148 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:22.323118    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-jtc7s"]
Sep 16 21:30:23.730626 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:23.730590    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qt792"]
Sep 16 21:30:23.731024 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:23.730785    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-qt792" podUID="a0946411-948d-4d20-adb5-3cc629fd7dd5" containerName="authorino" containerID="cri-o://38d24e8ef013486cd8553d8f3add4eaa78aee44e5f0082c09644370b494ca5ca" gracePeriod=30
Sep 16 21:30:23.943014 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:23.942981    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d7234df-14a4-43df-83e4-36d6dc71c067" path="/var/lib/kubelet/pods/7d7234df-14a4-43df-83e4-36d6dc71c067/volumes"
Sep 16 21:30:24.306429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:24.306397    2567 generic.go:356] "Generic (PLEG): container finished" podID="a0946411-948d-4d20-adb5-3cc629fd7dd5" containerID="38d24e8ef013486cd8553d8f3add4eaa78aee44e5f0082c09644370b494ca5ca" exitCode=0
Sep 16 21:30:24.306618 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:24.306469    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qt792" event={"ID":"a0946411-948d-4d20-adb5-3cc629fd7dd5","Type":"ContainerDied","Data":"38d24e8ef013486cd8553d8f3add4eaa78aee44e5f0082c09644370b494ca5ca"}
Sep 16 21:30:24.522556 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:24.522516    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qt792"
Sep 16 21:30:24.642615 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:24.642515    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2bd8\" (UniqueName: \"kubernetes.io/projected/a0946411-948d-4d20-adb5-3cc629fd7dd5-kube-api-access-t2bd8\") pod \"a0946411-948d-4d20-adb5-3cc629fd7dd5\" (UID: \"a0946411-948d-4d20-adb5-3cc629fd7dd5\") "
Sep 16 21:30:24.644728 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:24.644696    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0946411-948d-4d20-adb5-3cc629fd7dd5-kube-api-access-t2bd8" (OuterVolumeSpecName: "kube-api-access-t2bd8") pod "a0946411-948d-4d20-adb5-3cc629fd7dd5" (UID: "a0946411-948d-4d20-adb5-3cc629fd7dd5"). InnerVolumeSpecName "kube-api-access-t2bd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:30:24.743992 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:24.743952    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2bd8\" (UniqueName: \"kubernetes.io/projected/a0946411-948d-4d20-adb5-3cc629fd7dd5-kube-api-access-t2bd8\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:30:25.312366 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:25.312335    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-qt792"
Sep 16 21:30:25.312366 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:25.312352    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-qt792" event={"ID":"a0946411-948d-4d20-adb5-3cc629fd7dd5","Type":"ContainerDied","Data":"f12ce017c7e62312dce9c690de734f4ada5a5849df301e0ca6c73c6e38ee4b54"}
Sep 16 21:30:25.312641 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:25.312405    2567 scope.go:117] "RemoveContainer" containerID="38d24e8ef013486cd8553d8f3add4eaa78aee44e5f0082c09644370b494ca5ca"
Sep 16 21:30:25.332388 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:25.332356    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qt792"]
Sep 16 21:30:25.335363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:25.335336    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-qt792"]
Sep 16 21:30:25.941415 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:25.941377    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0946411-948d-4d20-adb5-3cc629fd7dd5" path="/var/lib/kubelet/pods/a0946411-948d-4d20-adb5-3cc629fd7dd5/volumes"
Sep 16 21:30:28.616199 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.616158    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:28.644613 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.644580    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-669bd445f5-wvqjh"]
Sep 16 21:30:28.644963 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.644950    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0946411-948d-4d20-adb5-3cc629fd7dd5" containerName="authorino"
Sep 16 21:30:28.645018 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.644965    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0946411-948d-4d20-adb5-3cc629fd7dd5" containerName="authorino"
Sep 16 21:30:28.645066 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.645041    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0946411-948d-4d20-adb5-3cc629fd7dd5" containerName="authorino"
Sep 16 21:30:28.649625 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.649601    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:28.652171 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.652138    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 21:30:28.652332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.652186    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 21:30:28.652332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.652203    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 21:30:28.652332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.652285    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nfgqp\""
Sep 16 21:30:28.660863 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.660833    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-669bd445f5-wvqjh"]
Sep 16 21:30:28.716097 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.716042    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:28.780796 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.780756    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-metrics-certs\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:28.781014 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.780840    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wprc4\" (UniqueName: \"kubernetes.io/projected/a620b18c-8059-4a65-90e6-fd2f0302c06d-kube-api-access-wprc4\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:28.781014 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.780868    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-webhook-cert\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:28.818079 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.818036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:28.830337 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.830305    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d8855c556-74bwf"]
Sep 16 21:30:28.833606 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.833589    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:28.844504 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.844480    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8855c556-74bwf"]
Sep 16 21:30:28.882204 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.882112    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprc4\" (UniqueName: \"kubernetes.io/projected/a620b18c-8059-4a65-90e6-fd2f0302c06d-kube-api-access-wprc4\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:28.882204 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.882157    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-webhook-cert\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:28.882448 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.882224    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-metrics-certs\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:28.882448 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:30:28.882322    2567 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 16 21:30:28.882448 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:30:28.882385    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-webhook-cert podName:a620b18c-8059-4a65-90e6-fd2f0302c06d nodeName:}" failed. No retries permitted until 2026-09-16 21:30:29.382369769 +0000 UTC m=+516.030781753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-webhook-cert") pod "maas-controller-669bd445f5-wvqjh" (UID: "a620b18c-8059-4a65-90e6-fd2f0302c06d") : secret "maas-controller-webhook-cert" not found
Sep 16 21:30:28.884791 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.884766    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-metrics-certs\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:28.890434 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.890404    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:28.893081 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.893060    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprc4\" (UniqueName: \"kubernetes.io/projected/a620b18c-8059-4a65-90e6-fd2f0302c06d-kube-api-access-wprc4\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:28.981982 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.981942    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:28.982949 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.982924    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9kb\" (UniqueName: \"kubernetes.io/projected/95406f41-7566-4fc1-a4c5-13ec5eddebef-kube-api-access-gr9kb\") pod \"maas-controller-6d8855c556-74bwf\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") " pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:28.983011 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.982979    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-webhook-cert\") pod \"maas-controller-6d8855c556-74bwf\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") " pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:28.983087 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:28.983068    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-metrics-certs\") pod \"maas-controller-6d8855c556-74bwf\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") " pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:29.083945 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.083870    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-webhook-cert\") pod \"maas-controller-6d8855c556-74bwf\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") " pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:29.084093 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.083977    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-metrics-certs\") pod \"maas-controller-6d8855c556-74bwf\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") " pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:29.084093 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.084031    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9kb\" (UniqueName: \"kubernetes.io/projected/95406f41-7566-4fc1-a4c5-13ec5eddebef-kube-api-access-gr9kb\") pod \"maas-controller-6d8855c556-74bwf\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") " pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:29.086503 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.086470    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-webhook-cert\") pod \"maas-controller-6d8855c556-74bwf\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") " pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:29.086617 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.086526    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-metrics-certs\") pod \"maas-controller-6d8855c556-74bwf\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") " pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:29.092244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.092222    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9kb\" (UniqueName: \"kubernetes.io/projected/95406f41-7566-4fc1-a4c5-13ec5eddebef-kube-api-access-gr9kb\") pod \"maas-controller-6d8855c556-74bwf\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") " pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:29.144391 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.144293    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:29.386420 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.386383    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-webhook-cert\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:29.388970 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.388943    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-webhook-cert\") pod \"maas-controller-669bd445f5-wvqjh\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") " pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:29.415348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.415220    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8855c556-74bwf"]
Sep 16 21:30:29.418019 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:30:29.417989    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95406f41_7566_4fc1_a4c5_13ec5eddebef.slice/crio-7c12c9b5b9f0c448259275b50edd365bf6e3318f5aee45e538e10f5d3afdc35d WatchSource:0}: Error finding container 7c12c9b5b9f0c448259275b50edd365bf6e3318f5aee45e538e10f5d3afdc35d: Status 404 returned error can't find the container with id 7c12c9b5b9f0c448259275b50edd365bf6e3318f5aee45e538e10f5d3afdc35d
Sep 16 21:30:29.562298 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.562261    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:29.835231 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:29.835053    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-669bd445f5-wvqjh"]
Sep 16 21:30:29.837783 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:30:29.837756    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda620b18c_8059_4a65_90e6_fd2f0302c06d.slice/crio-938dc8c693631ef9bad5d67e537606a6126c8211fc9faf3a53b870278af74eda WatchSource:0}: Error finding container 938dc8c693631ef9bad5d67e537606a6126c8211fc9faf3a53b870278af74eda: Status 404 returned error can't find the container with id 938dc8c693631ef9bad5d67e537606a6126c8211fc9faf3a53b870278af74eda
Sep 16 21:30:30.343523 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:30.343423    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-669bd445f5-wvqjh" event={"ID":"a620b18c-8059-4a65-90e6-fd2f0302c06d","Type":"ContainerStarted","Data":"938dc8c693631ef9bad5d67e537606a6126c8211fc9faf3a53b870278af74eda"}
Sep 16 21:30:30.346277 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:30.346236    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-74bwf" event={"ID":"95406f41-7566-4fc1-a4c5-13ec5eddebef","Type":"ContainerStarted","Data":"7c12c9b5b9f0c448259275b50edd365bf6e3318f5aee45e538e10f5d3afdc35d"}
Sep 16 21:30:32.298093 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:32.298059    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:32.357556 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:32.357447    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-669bd445f5-wvqjh" event={"ID":"a620b18c-8059-4a65-90e6-fd2f0302c06d","Type":"ContainerStarted","Data":"5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d"}
Sep 16 21:30:32.357746 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:32.357582    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:32.358980 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:32.358954    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-74bwf" event={"ID":"95406f41-7566-4fc1-a4c5-13ec5eddebef","Type":"ContainerStarted","Data":"b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd"}
Sep 16 21:30:32.359086 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:32.359010    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:32.377245 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:32.377191    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-669bd445f5-wvqjh" podStartSLOduration=2.116868718 podStartE2EDuration="4.377175647s" totalImagesPullingTime="2.260306929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:30:32.375820455 +0000 UTC m=+519.024232443" watchObservedRunningTime="2026-09-16 21:30:32.377175647 +0000 UTC m=+519.025587634"
Sep 16 21:30:32.382593 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:32.382564    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:32.394563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:32.394504    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d8855c556-74bwf" podStartSLOduration=1.725125462 podStartE2EDuration="4.3944799s" totalImagesPullingTime="2.669354438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:30:32.393765638 +0000 UTC m=+519.042177626" watchObservedRunningTime="2026-09-16 21:30:32.3944799 +0000 UTC m=+519.042891890"
Sep 16 21:30:43.366774 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.366739    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:43.367193 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.366920    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:30:43.478964 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.478933    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-669bd445f5-wvqjh"]
Sep 16 21:30:43.479165 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.479120    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-669bd445f5-wvqjh" podUID="a620b18c-8059-4a65-90e6-fd2f0302c06d" containerName="manager" containerID="cri-o://5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d" gracePeriod=30
Sep 16 21:30:43.729372 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.729345    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:43.806133 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.806094    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wprc4\" (UniqueName: \"kubernetes.io/projected/a620b18c-8059-4a65-90e6-fd2f0302c06d-kube-api-access-wprc4\") pod \"a620b18c-8059-4a65-90e6-fd2f0302c06d\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") "
Sep 16 21:30:43.806312 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.806151    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-metrics-certs\") pod \"a620b18c-8059-4a65-90e6-fd2f0302c06d\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") "
Sep 16 21:30:43.806312 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.806183    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-webhook-cert\") pod \"a620b18c-8059-4a65-90e6-fd2f0302c06d\" (UID: \"a620b18c-8059-4a65-90e6-fd2f0302c06d\") "
Sep 16 21:30:43.808424 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.808389    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a620b18c-8059-4a65-90e6-fd2f0302c06d-kube-api-access-wprc4" (OuterVolumeSpecName: "kube-api-access-wprc4") pod "a620b18c-8059-4a65-90e6-fd2f0302c06d" (UID: "a620b18c-8059-4a65-90e6-fd2f0302c06d"). InnerVolumeSpecName "kube-api-access-wprc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:30:43.808542 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.808457    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "a620b18c-8059-4a65-90e6-fd2f0302c06d" (UID: "a620b18c-8059-4a65-90e6-fd2f0302c06d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:30:43.808586 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.808535    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a620b18c-8059-4a65-90e6-fd2f0302c06d" (UID: "a620b18c-8059-4a65-90e6-fd2f0302c06d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:30:43.906989 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.906889    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wprc4\" (UniqueName: \"kubernetes.io/projected/a620b18c-8059-4a65-90e6-fd2f0302c06d-kube-api-access-wprc4\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:30:43.906989 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.906938    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:30:43.906989 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:43.906948    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a620b18c-8059-4a65-90e6-fd2f0302c06d-webhook-cert\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:30:44.416755 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:44.416713    2567 generic.go:356] "Generic (PLEG): container finished" podID="a620b18c-8059-4a65-90e6-fd2f0302c06d" containerID="5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d" exitCode=0
Sep 16 21:30:44.417208 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:44.416780    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-669bd445f5-wvqjh"
Sep 16 21:30:44.417208 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:44.416799    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-669bd445f5-wvqjh" event={"ID":"a620b18c-8059-4a65-90e6-fd2f0302c06d","Type":"ContainerDied","Data":"5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d"}
Sep 16 21:30:44.417208 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:44.416880    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-669bd445f5-wvqjh" event={"ID":"a620b18c-8059-4a65-90e6-fd2f0302c06d","Type":"ContainerDied","Data":"938dc8c693631ef9bad5d67e537606a6126c8211fc9faf3a53b870278af74eda"}
Sep 16 21:30:44.417208 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:44.416922    2567 scope.go:117] "RemoveContainer" containerID="5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d"
Sep 16 21:30:44.425473 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:44.425453    2567 scope.go:117] "RemoveContainer" containerID="5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d"
Sep 16 21:30:44.425767 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:30:44.425747    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d\": container with ID starting with 5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d not found: ID does not exist" containerID="5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d"
Sep 16 21:30:44.425815 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:44.425777    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d"} err="failed to get container status \"5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d\": rpc error: code = NotFound desc = could not find container \"5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d\": container with ID starting with 5e3c20aac15e777166853554006fc55ffe1c45912c2c49029242b42c7c39e95d not found: ID does not exist"
Sep 16 21:30:44.433986 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:44.433959    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-669bd445f5-wvqjh"]
Sep 16 21:30:44.437509 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:44.437481    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-669bd445f5-wvqjh"]
Sep 16 21:30:45.941567 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:45.941533    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a620b18c-8059-4a65-90e6-fd2f0302c06d" path="/var/lib/kubelet/pods/a620b18c-8059-4a65-90e6-fd2f0302c06d/volumes"
Sep 16 21:30:54.713825 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:54.713787    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:30:54.769979 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:30:54.769943    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:01.453200 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:01.453154    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:01.565295 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:01.565253    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:02.868732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:02.868696    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:02.961277 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:02.961236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:03.003326 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.003289    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"]
Sep 16 21:31:03.003678 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.003664    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a620b18c-8059-4a65-90e6-fd2f0302c06d" containerName="manager"
Sep 16 21:31:03.003722 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.003680    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a620b18c-8059-4a65-90e6-fd2f0302c06d" containerName="manager"
Sep 16 21:31:03.003772 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.003762    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a620b18c-8059-4a65-90e6-fd2f0302c06d" containerName="manager"
Sep 16 21:31:03.010142 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.010116    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.013619 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.013589    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 21:31:03.013619 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.013609    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lnht4\""
Sep 16 21:31:03.013833 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.013615    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 21:31:03.019288 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.019264    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"]
Sep 16 21:31:03.039428 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.039386    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:03.066422 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.066377    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-metrics-certs\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.066422 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.066430    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-maas-api-tls\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.066644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.066449    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34069190-3e33-4cbe-aefd-87de44dc3a5c-openshift-service-ca\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.066644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.066516    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njg8s\" (UniqueName: \"kubernetes.io/projected/34069190-3e33-4cbe-aefd-87de44dc3a5c-kube-api-access-njg8s\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.168025 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.167926    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-metrics-certs\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.168025 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.167979    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-maas-api-tls\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.168025 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.168001    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34069190-3e33-4cbe-aefd-87de44dc3a5c-openshift-service-ca\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.168317 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.168032    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njg8s\" (UniqueName: \"kubernetes.io/projected/34069190-3e33-4cbe-aefd-87de44dc3a5c-kube-api-access-njg8s\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.168317 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:31:03.168056    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 16 21:31:03.168317 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:31:03.168143    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-metrics-certs podName:34069190-3e33-4cbe-aefd-87de44dc3a5c nodeName:}" failed. No retries permitted until 2026-09-16 21:31:03.668119682 +0000 UTC m=+550.316531652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-metrics-certs") pod "maas-api-7d747b4f9d-jdfdt" (UID: "34069190-3e33-4cbe-aefd-87de44dc3a5c") : secret "maas-api-metrics-tls" not found
Sep 16 21:31:03.168805 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.168785    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34069190-3e33-4cbe-aefd-87de44dc3a5c-openshift-service-ca\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.170635 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.170615    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-maas-api-tls\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.179524 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.179496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njg8s\" (UniqueName: \"kubernetes.io/projected/34069190-3e33-4cbe-aefd-87de44dc3a5c-kube-api-access-njg8s\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.201033 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.200992    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:03.302281 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.302241    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:03.403031 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.403001    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:03.475136 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.475056    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"]
Sep 16 21:31:03.479157 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.479140    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:03.481702 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.481681    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 21:31:03.481840 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.481682    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-nrlmm\""
Sep 16 21:31:03.490329 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.490297    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"]
Sep 16 21:31:03.492359 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.492329    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:03.571225 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.571187    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtxh\" (UniqueName: \"kubernetes.io/projected/3d3cee08-b372-4481-a06d-a0325f07b0e1-kube-api-access-shtxh\") pod \"payload-pre-processing-59fbc47bf8-454lz\" (UID: \"3d3cee08-b372-4481-a06d-a0325f07b0e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:03.571416 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.571268    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3cee08-b372-4481-a06d-a0325f07b0e1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-454lz\" (UID: \"3d3cee08-b372-4481-a06d-a0325f07b0e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:03.672691 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.672654    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtxh\" (UniqueName: \"kubernetes.io/projected/3d3cee08-b372-4481-a06d-a0325f07b0e1-kube-api-access-shtxh\") pod \"payload-pre-processing-59fbc47bf8-454lz\" (UID: \"3d3cee08-b372-4481-a06d-a0325f07b0e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:03.672892 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.672723    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3cee08-b372-4481-a06d-a0325f07b0e1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-454lz\" (UID: \"3d3cee08-b372-4481-a06d-a0325f07b0e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:03.672892 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.672762    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-metrics-certs\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.673554 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.673533    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d3cee08-b372-4481-a06d-a0325f07b0e1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-454lz\" (UID: \"3d3cee08-b372-4481-a06d-a0325f07b0e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:03.675355 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.675328    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-metrics-certs\") pod \"maas-api-7d747b4f9d-jdfdt\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") " pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:03.681338 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.681314    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtxh\" (UniqueName: \"kubernetes.io/projected/3d3cee08-b372-4481-a06d-a0325f07b0e1-kube-api-access-shtxh\") pod \"payload-pre-processing-59fbc47bf8-454lz\" (UID: \"3d3cee08-b372-4481-a06d-a0325f07b0e1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:03.790838 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.790801    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:03.921542 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:03.921499    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:04.062364 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:04.062339    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"]
Sep 16 21:31:04.064587 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:31:04.064557    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3cee08_b372_4481_a06d_a0325f07b0e1.slice/crio-97c2b534c720f152b587cce9edc69270ccf0042c285c1a66d86e0457d5a459de WatchSource:0}: Error finding container 97c2b534c720f152b587cce9edc69270ccf0042c285c1a66d86e0457d5a459de: Status 404 returned error can't find the container with id 97c2b534c720f152b587cce9edc69270ccf0042c285c1a66d86e0457d5a459de
Sep 16 21:31:04.197968 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:04.197943    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"]
Sep 16 21:31:04.200515 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:31:04.200485    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34069190_3e33_4cbe_aefd_87de44dc3a5c.slice/crio-3ed7dcea472a9279aa67ad9e1bc3d4d7d07ed88c668328e89250041bc102533f WatchSource:0}: Error finding container 3ed7dcea472a9279aa67ad9e1bc3d4d7d07ed88c668328e89250041bc102533f: Status 404 returned error can't find the container with id 3ed7dcea472a9279aa67ad9e1bc3d4d7d07ed88c668328e89250041bc102533f
Sep 16 21:31:04.510929 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:04.510884    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz" event={"ID":"3d3cee08-b372-4481-a06d-a0325f07b0e1","Type":"ContainerStarted","Data":"97c2b534c720f152b587cce9edc69270ccf0042c285c1a66d86e0457d5a459de"}
Sep 16 21:31:04.512002 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:04.511980    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt" event={"ID":"34069190-3e33-4cbe-aefd-87de44dc3a5c","Type":"ContainerStarted","Data":"3ed7dcea472a9279aa67ad9e1bc3d4d7d07ed88c668328e89250041bc102533f"}
Sep 16 21:31:07.529813 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:07.529776    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt" event={"ID":"34069190-3e33-4cbe-aefd-87de44dc3a5c","Type":"ContainerStarted","Data":"e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53"}
Sep 16 21:31:07.530342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:07.529943    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:07.531464 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:07.531439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz" event={"ID":"3d3cee08-b372-4481-a06d-a0325f07b0e1","Type":"ContainerStarted","Data":"c50682d2e87f5faad59a768686b8c3e0ca9f5d17e44d9fc024d645b62e8f9790"}
Sep 16 21:31:07.531584 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:07.531505    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:07.547858 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:07.547806    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt" podStartSLOduration=2.826193278 podStartE2EDuration="5.547790225s" totalImagesPullingTime="2.721596947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:31:07.54633023 +0000 UTC m=+554.194742219" watchObservedRunningTime="2026-09-16 21:31:07.547790225 +0000 UTC m=+554.196202212"
Sep 16 21:31:07.561937 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:07.561865    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz" podStartSLOduration=1.705269554 podStartE2EDuration="4.561848211s" totalImagesPullingTime="2.856578657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:31:07.560962822 +0000 UTC m=+554.209374811" watchObservedRunningTime="2026-09-16 21:31:07.561848211 +0000 UTC m=+554.210260199"
Sep 16 21:31:13.542039 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:13.542008    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 21:31:14.578193 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:14.578153    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:14.700919 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:14.700867    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:18.538621 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:18.538586    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-454lz"
Sep 16 21:31:22.891026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:22.890993    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"]
Sep 16 21:31:22.894853 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:22.894829    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:22.897118 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:22.897095    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pjb85\""
Sep 16 21:31:22.898102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:22.898078    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 21:31:22.898242 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:22.898078    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 16 21:31:22.898242 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:22.898222    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 21:31:22.910109 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:22.910083    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"]
Sep 16 21:31:22.920446 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:22.920413    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:23.047626 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.047590    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kt2m\" (UniqueName: \"kubernetes.io/projected/b27766ac-e448-487a-ad44-e3195975e93d-kube-api-access-2kt2m\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.047805 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.047652    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.047805 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.047673    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.047805 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.047702    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.047805 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.047755    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27766ac-e448-487a-ad44-e3195975e93d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.048018 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.047806    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.148606 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.148526    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.148606 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.148592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kt2m\" (UniqueName: \"kubernetes.io/projected/b27766ac-e448-487a-ad44-e3195975e93d-kube-api-access-2kt2m\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.148833 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.148631    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.148833 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.148645    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.148833 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.148661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.148833 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.148805    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27766ac-e448-487a-ad44-e3195975e93d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.149070 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.149047    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.149070 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.149065    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.149178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.149152    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.151070 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.151048    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b27766ac-e448-487a-ad44-e3195975e93d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.151211 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.151194    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b27766ac-e448-487a-ad44-e3195975e93d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.157025 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.157002    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kt2m\" (UniqueName: \"kubernetes.io/projected/b27766ac-e448-487a-ad44-e3195975e93d-kube-api-access-2kt2m\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg\" (UID: \"b27766ac-e448-487a-ad44-e3195975e93d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.205701 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.205659    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:23.478142 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.478117    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"]
Sep 16 21:31:23.479874 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:31:23.479849    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27766ac_e448_487a_ad44_e3195975e93d.slice/crio-d67441ceb175a90eb2058b1c3aee82cfd39244a728724550ad17de033385c247 WatchSource:0}: Error finding container d67441ceb175a90eb2058b1c3aee82cfd39244a728724550ad17de033385c247: Status 404 returned error can't find the container with id d67441ceb175a90eb2058b1c3aee82cfd39244a728724550ad17de033385c247
Sep 16 21:31:23.604917 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:23.604870    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg" event={"ID":"b27766ac-e448-487a-ad44-e3195975e93d","Type":"ContainerStarted","Data":"d67441ceb175a90eb2058b1c3aee82cfd39244a728724550ad17de033385c247"}
Sep 16 21:31:24.611307 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:24.611269    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg" event={"ID":"b27766ac-e448-487a-ad44-e3195975e93d","Type":"ContainerStarted","Data":"cb58bf7e402b42fe44e3a5fd7288552fb7b15abd94a20d2cbfc0f4b92fe2508e"}
Sep 16 21:31:24.627988 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:24.627927    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg" podStartSLOduration=1.625330666 podStartE2EDuration="2.627893829s" totalImagesPullingTime="1.002563163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:31:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:31:24.626945546 +0000 UTC m=+571.275357527" watchObservedRunningTime="2026-09-16 21:31:24.627893829 +0000 UTC m=+571.276305825"
Sep 16 21:31:30.885547 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:30.885497    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:33.206426 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:33.206383    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:33.206823 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:33.206447    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:33.219083 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:33.219053    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:33.663735 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:33.663704    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg"
Sep 16 21:31:51.084353 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.084320    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"]
Sep 16 21:31:51.087568 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.087548    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.090172 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.090143    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 21:31:51.097638 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.097615    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"]
Sep 16 21:31:51.116283 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.116253    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:31:51.180257 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.180224    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.180257 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.180258    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.180498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.180298    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.180498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.180366    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.180498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.180432    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwbf\" (UniqueName: \"kubernetes.io/projected/a69d8e80-4cd1-4422-913f-cd8ddad9e621-kube-api-access-skwbf\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.180498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.180476    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69d8e80-4cd1-4422-913f-cd8ddad9e621-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.280980 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.280939    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.281159 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.280994    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.281159 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.281058    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.281159 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.281089    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.281159 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.281125    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwbf\" (UniqueName: \"kubernetes.io/projected/a69d8e80-4cd1-4422-913f-cd8ddad9e621-kube-api-access-skwbf\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.281159 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.281154    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69d8e80-4cd1-4422-913f-cd8ddad9e621-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.281471 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.281414    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.281602 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.281528    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.281674 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.281628    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.283663 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.283631    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69d8e80-4cd1-4422-913f-cd8ddad9e621-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.283934 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.283891    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69d8e80-4cd1-4422-913f-cd8ddad9e621-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.289393 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.289369    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwbf\" (UniqueName: \"kubernetes.io/projected/a69d8e80-4cd1-4422-913f-cd8ddad9e621-kube-api-access-skwbf\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf\" (UID: \"a69d8e80-4cd1-4422-913f-cd8ddad9e621\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.399592 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.399499    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:31:51.668562 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.668536    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"]
Sep 16 21:31:51.670490 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:31:51.670464    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69d8e80_4cd1_4422_913f_cd8ddad9e621.slice/crio-80ed0041bd0d5ac6d7e4c9d43a518f3a4bb8fde1a7a2f397f777e74e288c986f WatchSource:0}: Error finding container 80ed0041bd0d5ac6d7e4c9d43a518f3a4bb8fde1a7a2f397f777e74e288c986f: Status 404 returned error can't find the container with id 80ed0041bd0d5ac6d7e4c9d43a518f3a4bb8fde1a7a2f397f777e74e288c986f
Sep 16 21:31:51.747492 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:51.747455    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf" event={"ID":"a69d8e80-4cd1-4422-913f-cd8ddad9e621","Type":"ContainerStarted","Data":"80ed0041bd0d5ac6d7e4c9d43a518f3a4bb8fde1a7a2f397f777e74e288c986f"}
Sep 16 21:31:52.753294 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:52.753258    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf" event={"ID":"a69d8e80-4cd1-4422-913f-cd8ddad9e621","Type":"ContainerStarted","Data":"1d7a2b5177cf23e6c228a7e3b2ff45848210e2282d554c1e5483bff49b9ffee4"}
Sep 16 21:31:52.773342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:52.773289    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf" podStartSLOduration=1.56688167 podStartE2EDuration="1.773272958s" totalImagesPullingTime="206.391288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:31:52.771346983 +0000 UTC m=+599.419758974" watchObservedRunningTime="2026-09-16 21:31:52.773272958 +0000 UTC m=+599.421684947"
Sep 16 21:31:53.801577 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:53.801546    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:31:53.805697 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:53.805673    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:31:58.173915 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:31:58.173872    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:32:01.400047 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:01.400003    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:32:01.402479 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:01.400115    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:32:01.412754 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:01.412724    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:32:01.809993 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:01.809963    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf"
Sep 16 21:32:04.970741 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:04.970701    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:32:11.087482 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.087450    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"]
Sep 16 21:32:11.090628 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.090610    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.092998 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.092979    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 16 21:32:11.104194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.104170    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"]
Sep 16 21:32:11.169388 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.169343    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.169549 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.169413    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldxf\" (UniqueName: \"kubernetes.io/projected/b546a798-80ed-4923-864f-2f7c12bc7800-kube-api-access-vldxf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.169549 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.169459    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.169549 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.169495    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b546a798-80ed-4923-864f-2f7c12bc7800-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.169653 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.169557    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.169691 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.169650    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.188283 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.188246    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:32:11.270632 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.270592    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.270632 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.270634    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b546a798-80ed-4923-864f-2f7c12bc7800-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.270930 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.270675    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.270930 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.270733    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.270930 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.270808    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.270930 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.270832    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldxf\" (UniqueName: \"kubernetes.io/projected/b546a798-80ed-4923-864f-2f7c12bc7800-kube-api-access-vldxf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.271220 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.271194    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.271363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.271214    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.271363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.271224    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.273453 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.273435    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b546a798-80ed-4923-864f-2f7c12bc7800-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.273862 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.273840    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b546a798-80ed-4923-864f-2f7c12bc7800-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.278631 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.278610    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldxf\" (UniqueName: \"kubernetes.io/projected/b546a798-80ed-4923-864f-2f7c12bc7800-kube-api-access-vldxf\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp\" (UID: \"b546a798-80ed-4923-864f-2f7c12bc7800\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.400789 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.400686    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:11.686926 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.684675    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"]
Sep 16 21:32:11.687893 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.687868    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:32:11.847301 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:11.847265    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp" event={"ID":"b546a798-80ed-4923-864f-2f7c12bc7800","Type":"ContainerStarted","Data":"99ef2cd05ec765004e37f04435d764a91bef24f87a77b952e2db120eb3eeec5b"}
Sep 16 21:32:12.853282 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:12.853242    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp" event={"ID":"b546a798-80ed-4923-864f-2f7c12bc7800","Type":"ContainerStarted","Data":"a7fccd9be1c00bd635047238343ff5734b79d6c1002ffd2f8897394411dcff5d"}
Sep 16 21:32:12.870643 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:12.870593    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp" podStartSLOduration=1.669412506 podStartE2EDuration="1.870576928s" totalImagesPullingTime="201.164422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:32:12.869832053 +0000 UTC m=+619.518244053" watchObservedRunningTime="2026-09-16 21:32:12.870576928 +0000 UTC m=+619.518988915"
Sep 16 21:32:20.986589 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:20.986508    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"]
Sep 16 21:32:20.990249 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:20.990226    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:20.992534 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:20.992513    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 16 21:32:21.002016 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.001990    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"]
Sep 16 21:32:21.023434 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.023403    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:32:21.156659 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.156619    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.156659 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.156656    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.156945 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.156676    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.156945 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.156694    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5zl\" (UniqueName: \"kubernetes.io/projected/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-kube-api-access-dw5zl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.156945 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.156834    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.156945 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.156877    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.257527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.257490    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.257716 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.257546    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.257716 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.257575    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.257716 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.257599    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.257716 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.257622    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5zl\" (UniqueName: \"kubernetes.io/projected/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-kube-api-access-dw5zl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.258039 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.257718    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.258096 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.258062    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.258153 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.258116    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.258213 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.258153    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.259881 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.259860    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.260295 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.260276    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.274159 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.274133    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5zl\" (UniqueName: \"kubernetes.io/projected/9a8b35e1-3a52-453b-ad21-d0e03ebc7db3-kube-api-access-dw5zl\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vf42p\" (UID: \"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.304177 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.304141    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:21.401497 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.401464    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:21.401655 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.401523    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:21.414162 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.414136    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:21.575203 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.575178    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"]
Sep 16 21:32:21.577310 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:32:21.577287    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8b35e1_3a52_453b_ad21_d0e03ebc7db3.slice/crio-2d7e709ffa240abb3a89dcd631f201a002e11e4a6d2f7dff2a035b057300c227 WatchSource:0}: Error finding container 2d7e709ffa240abb3a89dcd631f201a002e11e4a6d2f7dff2a035b057300c227: Status 404 returned error can't find the container with id 2d7e709ffa240abb3a89dcd631f201a002e11e4a6d2f7dff2a035b057300c227
Sep 16 21:32:21.904785 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.904744    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p" event={"ID":"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3","Type":"ContainerStarted","Data":"1f55ad4f0c4de9917b8868ee5affff571431169b23ad2c79c3773354a34dc8e8"}
Sep 16 21:32:21.904999 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.904815    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p" event={"ID":"9a8b35e1-3a52-453b-ad21-d0e03ebc7db3","Type":"ContainerStarted","Data":"2d7e709ffa240abb3a89dcd631f201a002e11e4a6d2f7dff2a035b057300c227"}
Sep 16 21:32:21.916684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.916660    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp"
Sep 16 21:32:21.924209 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:21.924167    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p" podStartSLOduration=1.7079230779999999 podStartE2EDuration="1.92415257s" totalImagesPullingTime="216.229492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:32:21.922339946 +0000 UTC m=+628.570751946" watchObservedRunningTime="2026-09-16 21:32:21.92415257 +0000 UTC m=+628.572564558"
Sep 16 21:32:24.790943 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:24.790871    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:32:31.305036 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:31.304999    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:31.305457 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:31.305055    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:31.317614 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:31.317586    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:31.962446 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:31.962416    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vf42p"
Sep 16 21:32:59.025338 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:32:59.025296    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:37.135026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:37.134989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:37.224811 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:37.224774    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:38.282190 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:38.282157    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:38.412764 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:38.412730    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:47.466309 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:47.466216    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:54.834871 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:54.834817    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:54.897050 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:54.897016    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:54.964125 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:54.964087    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:55.209327 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:55.209236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:55.358817 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:55.358783    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:55.456164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:55.456124    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:55.552170 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:55.552137    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:55.652256 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:55.652187    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:56.286382 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:56.286339    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:56.386861 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:56.386827    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:56.739514 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:56.739425    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:58.635669 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:58.635633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:58.785782 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:58.785744    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:59.164228 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.164196    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:59.296749 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.296708    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:33:59.929995 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.929960    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"]
Sep 16 21:33:59.933762 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.933737    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:33:59.938979 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.938956    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e5568d-serving-cert\""
Sep 16 21:33:59.946271 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.946247    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"]
Sep 16 21:33:59.956801 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.956763    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb04fa2b-2476-4513-bafe-5ab497525b31-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:33:59.956983 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.956862    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmcl\" (UniqueName: \"kubernetes.io/projected/eb04fa2b-2476-4513-bafe-5ab497525b31-kube-api-access-vcmcl\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:33:59.956983 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.956927    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-metrics-certs\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:33:59.956983 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.956974    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:33:59.975544 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:33:59.975429    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:00.058007 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.057969    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmcl\" (UniqueName: \"kubernetes.io/projected/eb04fa2b-2476-4513-bafe-5ab497525b31-kube-api-access-vcmcl\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:00.058208 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.058012    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-metrics-certs\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:00.058208 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.058053    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:00.058208 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:34:00.058189    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-serving-cert: secret "maas-api-e2e-inf-a-e5568d-serving-cert" not found
Sep 16 21:34:00.058387 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:34:00.058251    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-maas-api-tls podName:eb04fa2b-2476-4513-bafe-5ab497525b31 nodeName:}" failed. No retries permitted until 2026-09-16 21:34:00.558230642 +0000 UTC m=+727.206642609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-maas-api-tls") pod "maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv" (UID: "eb04fa2b-2476-4513-bafe-5ab497525b31") : secret "maas-api-e2e-inf-a-e5568d-serving-cert" not found
Sep 16 21:34:00.058387 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.058271    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb04fa2b-2476-4513-bafe-5ab497525b31-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:00.058883 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.058857    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb04fa2b-2476-4513-bafe-5ab497525b31-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:00.060516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.060496    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-metrics-certs\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:00.068890 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.068867    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmcl\" (UniqueName: \"kubernetes.io/projected/eb04fa2b-2476-4513-bafe-5ab497525b31-kube-api-access-vcmcl\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:00.076396 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.076366    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:00.399522 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.399491    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:00.562977 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.562935    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:00.565467 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.565442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-maas-api-tls\") pod \"maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:00.655645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.655550    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:00.748645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.748613    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:00.846040 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:00.846001    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:01.133616 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.133592    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"]
Sep 16 21:34:01.135718 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:34:01.135680    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb04fa2b_2476_4513_bafe_5ab497525b31.slice/crio-719b375f4f3db405b6af7b29eefc3870f6691290d22d6bb7ed58ea1a03201631 WatchSource:0}: Error finding container 719b375f4f3db405b6af7b29eefc3870f6691290d22d6bb7ed58ea1a03201631: Status 404 returned error can't find the container with id 719b375f4f3db405b6af7b29eefc3870f6691290d22d6bb7ed58ea1a03201631
Sep 16 21:34:01.145443 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.145409    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"]
Sep 16 21:34:01.152132 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.152101    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.159666 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.158957    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"]
Sep 16 21:34:01.159666 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.159111    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-35f5fd-serving-cert\""
Sep 16 21:34:01.212946 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.212877    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:01.269959 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.269895    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cncl\" (UniqueName: \"kubernetes.io/projected/220bf348-bef5-4886-8dcb-bd7ea4704222-kube-api-access-8cncl\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.270136 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.270016    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-maas-api-tls\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.270136 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.270089    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-metrics-certs\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.270224 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.270185    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bf348-bef5-4886-8dcb-bd7ea4704222-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.355176 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.355141    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:01.371318 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.371280    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-metrics-certs\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.371516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.371336    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bf348-bef5-4886-8dcb-bd7ea4704222-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.371516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.371359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cncl\" (UniqueName: \"kubernetes.io/projected/220bf348-bef5-4886-8dcb-bd7ea4704222-kube-api-access-8cncl\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.371516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.371388    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-maas-api-tls\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.372206 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.372178    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bf348-bef5-4886-8dcb-bd7ea4704222-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.373992 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.373962    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-maas-api-tls\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.373992 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.373985    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-metrics-certs\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.396652 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.396537    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv" event={"ID":"eb04fa2b-2476-4513-bafe-5ab497525b31","Type":"ContainerStarted","Data":"719b375f4f3db405b6af7b29eefc3870f6691290d22d6bb7ed58ea1a03201631"}
Sep 16 21:34:01.397030 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.396998    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cncl\" (UniqueName: \"kubernetes.io/projected/220bf348-bef5-4886-8dcb-bd7ea4704222-kube-api-access-8cncl\") pod \"maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.470174 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.470140    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:01.760523 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.760469    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"]
Sep 16 21:34:01.762814 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:34:01.762787    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220bf348_bef5_4886_8dcb_bd7ea4704222.slice/crio-b5242741ad6bdcf4ef1b9312d5beb713537ef0e76c4bd1241583d01c440d8913 WatchSource:0}: Error finding container b5242741ad6bdcf4ef1b9312d5beb713537ef0e76c4bd1241583d01c440d8913: Status 404 returned error can't find the container with id b5242741ad6bdcf4ef1b9312d5beb713537ef0e76c4bd1241583d01c440d8913
Sep 16 21:34:01.933075 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:01.933036    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:02.029715 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.029618    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:02.403406 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.403311    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv" event={"ID":"eb04fa2b-2476-4513-bafe-5ab497525b31","Type":"ContainerStarted","Data":"cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58"}
Sep 16 21:34:02.403841 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.403448    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:02.404961 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.404938    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q" event={"ID":"220bf348-bef5-4886-8dcb-bd7ea4704222","Type":"ContainerStarted","Data":"6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32"}
Sep 16 21:34:02.405070 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.404974    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q" event={"ID":"220bf348-bef5-4886-8dcb-bd7ea4704222","Type":"ContainerStarted","Data":"b5242741ad6bdcf4ef1b9312d5beb713537ef0e76c4bd1241583d01c440d8913"}
Sep 16 21:34:02.405070 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.404994    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:02.439426 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.439109    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv" podStartSLOduration=2.7278412100000002 podStartE2EDuration="3.439087258s" totalImagesPullingTime="711.246048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:02.436676056 +0000 UTC m=+729.085088045" watchObservedRunningTime="2026-09-16 21:34:02.439087258 +0000 UTC m=+729.087499250"
Sep 16 21:34:02.477288 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.477219    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q" podStartSLOduration=1.098933515 podStartE2EDuration="1.477198175s" totalImagesPullingTime="378.26466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:02.475125019 +0000 UTC m=+729.123537008" watchObservedRunningTime="2026-09-16 21:34:02.477198175 +0000 UTC m=+729.125610164"
Sep 16 21:34:02.592718 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.592685    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"]
Sep 16 21:34:02.597180 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.597156    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.602997 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:34:02.602966    2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-e2e-worker-w5-e047d8-serving-cert\" is forbidden: User \"system:node:ip-10-0-135-209.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-209.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-e047d8-serving-cert\"" type="*v1.Secret"
Sep 16 21:34:02.604403 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:34:02.604373    2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" is forbidden: User \"system:node:ip-10-0-135-209.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-135-209.ec2.internal' and this object" podUID="b453d27e-231b-4e5f-a1ff-219cfd63b1a2" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.634718 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.634692    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"]
Sep 16 21:34:02.683760 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.683667    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78rh7\" (UniqueName: \"kubernetes.io/projected/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-kube-api-access-78rh7\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.683760 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.683727    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.683982 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.683850    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.683982 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.683934    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-metrics-certs\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.721604 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.721571    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:02.784709 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.784669    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.784967 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.784736    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-metrics-certs\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.784967 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.784783    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78rh7\" (UniqueName: \"kubernetes.io/projected/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-kube-api-access-78rh7\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.784967 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.784822    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.785479 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.785451    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.787352 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.787322    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-metrics-certs\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.796499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.796471    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78rh7\" (UniqueName: \"kubernetes.io/projected/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-kube-api-access-78rh7\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:02.929712 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:02.929675    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:03.153797 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.153758    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:03.480258 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.480175    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:03.648627 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.648585    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:03.749629 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.749596    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:03.785066 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:34:03.785037    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 16 21:34:03.785243 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:34:03.785131    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-maas-api-tls podName:b453d27e-231b-4e5f-a1ff-219cfd63b1a2 nodeName:}" failed. No retries permitted until 2026-09-16 21:34:04.285111369 +0000 UTC m=+730.933523335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-maas-api-tls") pod "maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb" (UID: "b453d27e-231b-4e5f-a1ff-219cfd63b1a2") : failed to sync secret cache: timed out waiting for the condition
Sep 16 21:34:03.885138 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.885104    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-e047d8-serving-cert\""
Sep 16 21:34:03.927879 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.927847    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"]
Sep 16 21:34:03.931470 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.931455    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:03.936358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.936335    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-e047d8\""
Sep 16 21:34:03.936843 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.936826    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-e047d8-dockercfg-d69dp\""
Sep 16 21:34:03.956465 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.956429    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"]
Sep 16 21:34:03.990512 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.990481    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:03.997595 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.997566    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v\" (UID: \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:03.998088 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:03.997887    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstmj\" (UniqueName: \"kubernetes.io/projected/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-kube-api-access-nstmj\") pod \"payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v\" (UID: \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:04.099339 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.099259    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstmj\" (UniqueName: \"kubernetes.io/projected/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-kube-api-access-nstmj\") pod \"payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v\" (UID: \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:04.099484 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.099420    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v\" (UID: \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:04.100107 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.100089    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v\" (UID: \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:04.116239 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.116209    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstmj\" (UniqueName: \"kubernetes.io/projected/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-kube-api-access-nstmj\") pod \"payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v\" (UID: \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:04.243030 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.242993    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:04.300949 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.300725    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:04.306482 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.306457    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-maas-api-tls\") pod \"maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:04.407522 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.407428    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:04.476009 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.474270    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:04.531265 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.531238    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"]
Sep 16 21:34:04.533419 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:34:04.533393    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c95c3c_01a9_434e_81c6_b2c8706d39ee.slice/crio-2edb0d8a16415ea7d27e7a5c8b54f2d2e96fa4f6c5e510d43439f1b7a7108878 WatchSource:0}: Error finding container 2edb0d8a16415ea7d27e7a5c8b54f2d2e96fa4f6c5e510d43439f1b7a7108878: Status 404 returned error can't find the container with id 2edb0d8a16415ea7d27e7a5c8b54f2d2e96fa4f6c5e510d43439f1b7a7108878
Sep 16 21:34:04.687672 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:04.687548    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"]
Sep 16 21:34:05.424379 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:05.424280    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb" event={"ID":"b453d27e-231b-4e5f-a1ff-219cfd63b1a2","Type":"ContainerStarted","Data":"551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2"}
Sep 16 21:34:05.424379 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:05.424339    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb" event={"ID":"b453d27e-231b-4e5f-a1ff-219cfd63b1a2","Type":"ContainerStarted","Data":"864968d5b253c234b4053aa0c0684b9230a187e9a809e5c5f9617a812c02bbf5"}
Sep 16 21:34:05.424629 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:05.424470    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:05.426085 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:05.426060    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v" event={"ID":"a2c95c3c-01a9-434e-81c6-b2c8706d39ee","Type":"ContainerStarted","Data":"13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170"}
Sep 16 21:34:05.426200 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:05.426099    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v" event={"ID":"a2c95c3c-01a9-434e-81c6-b2c8706d39ee","Type":"ContainerStarted","Data":"2edb0d8a16415ea7d27e7a5c8b54f2d2e96fa4f6c5e510d43439f1b7a7108878"}
Sep 16 21:34:05.426200 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:05.426120    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:05.455896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:05.455841    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb" podStartSLOduration=3.094474181 podStartE2EDuration="3.455824468s" totalImagesPullingTime="361.350287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:05.454453106 +0000 UTC m=+732.102865094" watchObservedRunningTime="2026-09-16 21:34:05.455824468 +0000 UTC m=+732.104236456"
Sep 16 21:34:05.493567 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:05.493505    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v" podStartSLOduration=2.493483446 podStartE2EDuration="2.493483446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:05.489851115 +0000 UTC m=+732.138263103" watchObservedRunningTime="2026-09-16 21:34:05.493483446 +0000 UTC m=+732.141895434"
Sep 16 21:34:05.611994 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:05.611958    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:06.251290 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:06.251250    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:06.508046 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:06.507957    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:06.600069 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:06.600035    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:06.699069 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:06.699035    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:06.907872 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:06.907780    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:07.149667 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:07.149626    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:07.342482 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:07.342436    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:07.464114 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:07.464080    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:08.341347 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.341300    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"]
Sep 16 21:34:08.345118 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.345096    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.348459 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.348437    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-ddbefa-serving-cert\""
Sep 16 21:34:08.359513 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.359484    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"]
Sep 16 21:34:08.389050 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.389017    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:08.416600 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.416571    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:34:08.417073 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.417053    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:34:08.439924 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.439871    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-metrics-certs\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.440083 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.439960    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fef8b-dc67-42c0-a80d-d381375dc03f-openshift-service-ca\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.440083 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.439990    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mg4z\" (UniqueName: \"kubernetes.io/projected/dc7fef8b-dc67-42c0-a80d-d381375dc03f-kube-api-access-7mg4z\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.440083 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.440030    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-maas-api-tls\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.487113 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.487081    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:08.540577 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.540544    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-maas-api-tls\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.540766 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.540663    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-metrics-certs\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.541320 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.540852    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fef8b-dc67-42c0-a80d-d381375dc03f-openshift-service-ca\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.541320 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.540984    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mg4z\" (UniqueName: \"kubernetes.io/projected/dc7fef8b-dc67-42c0-a80d-d381375dc03f-kube-api-access-7mg4z\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.541643 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.541622    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fef8b-dc67-42c0-a80d-d381375dc03f-openshift-service-ca\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.543247 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.543228    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-metrics-certs\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.543340 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.543255    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-maas-api-tls\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.559816 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.559780    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mg4z\" (UniqueName: \"kubernetes.io/projected/dc7fef8b-dc67-42c0-a80d-d381375dc03f-kube-api-access-7mg4z\") pod \"maas-api-e2e-autores-ddbefa-76555c5c8-nxk52\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.583195 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.583156    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:08.657962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.657835    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:08.966655 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:08.966630    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"]
Sep 16 21:34:08.968563 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:34:08.968540    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7fef8b_dc67_42c0_a80d_d381375dc03f.slice/crio-0e38c12066f6325a15d97af1011f1391a9b532bcdaf89208850203996ab245cb WatchSource:0}: Error finding container 0e38c12066f6325a15d97af1011f1391a9b532bcdaf89208850203996ab245cb: Status 404 returned error can't find the container with id 0e38c12066f6325a15d97af1011f1391a9b532bcdaf89208850203996ab245cb
Sep 16 21:34:09.134990 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:09.134956    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:09.327656 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:09.327621    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:09.463479 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:09.463444    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52" event={"ID":"dc7fef8b-dc67-42c0-a80d-d381375dc03f","Type":"ContainerStarted","Data":"90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b"}
Sep 16 21:34:09.463856 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:09.463517    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52" event={"ID":"dc7fef8b-dc67-42c0-a80d-d381375dc03f","Type":"ContainerStarted","Data":"0e38c12066f6325a15d97af1011f1391a9b532bcdaf89208850203996ab245cb"}
Sep 16 21:34:09.463856 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:09.463552    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:09.575311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:09.575271    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:09.819516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:09.819478    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:10.424477 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:10.424441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:11.437477 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:11.437449    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:34:11.470330 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:11.470285    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52" podStartSLOduration=3.107351593 podStartE2EDuration="3.47026827s" totalImagesPullingTime="362.916677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:09.510556637 +0000 UTC m=+736.158968624" watchObservedRunningTime="2026-09-16 21:34:11.47026827 +0000 UTC m=+738.118680257"
Sep 16 21:34:11.694171 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:11.694083    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:12.784970 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:12.784934    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:13.446050 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:13.446013    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:13.790242 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:13.790203    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:14.054598 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:14.054509    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:14.347336 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:14.347249    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:14.596731 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:14.596692    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:14.763406 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:14.763371    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:14.881710 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:14.881676    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:15.474738 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:15.474711    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:34:15.906945 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:15.906912    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"]
Sep 16 21:34:15.910612 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:15.910593    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:15.914935 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:15.914897    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e078853-serving-cert\""
Sep 16 21:34:15.934202 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:15.934177    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"]
Sep 16 21:34:16.004734 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.004700    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:16.012362 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.012335    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bkd\" (UniqueName: \"kubernetes.io/projected/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-kube-api-access-p4bkd\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.012525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.012382    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-metrics-certs\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.012525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.012412    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-maas-api-tls\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.012525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.012513    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-openshift-service-ca\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.113352 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.113320    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bkd\" (UniqueName: \"kubernetes.io/projected/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-kube-api-access-p4bkd\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.113562 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.113390    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-metrics-certs\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.113562 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.113441    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-maas-api-tls\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.113562 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:34:16.113556    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-serving-cert: secret "maas-api-e2e-ait-0e078853-serving-cert" not found
Sep 16 21:34:16.113730 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:34:16.113623    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-maas-api-tls podName:7b467dc4-35a9-45f1-9bd2-b630eaceb9e0 nodeName:}" failed. No retries permitted until 2026-09-16 21:34:16.613602341 +0000 UTC m=+743.262014313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-maas-api-tls") pod "maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl" (UID: "7b467dc4-35a9-45f1-9bd2-b630eaceb9e0") : secret "maas-api-e2e-ait-0e078853-serving-cert" not found
Sep 16 21:34:16.113730 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.113661    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-openshift-service-ca\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.114356 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.114328    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-openshift-service-ca\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.116187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.116157    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-metrics-certs\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.132572 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.132549    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bkd\" (UniqueName: \"kubernetes.io/projected/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-kube-api-access-p4bkd\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.136193 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.136169    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:16.257122 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.257086    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:16.434487 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.434459    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:34:16.619214 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.619125    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-maas-api-tls\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.621606 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.621583    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-maas-api-tls\") pod \"maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.821711 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.821678    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:16.958563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:16.958534    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:17.084186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:17.084152    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:17.103950 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:17.103926    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"]
Sep 16 21:34:17.106038 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:34:17.106014    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b467dc4_35a9_45f1_9bd2_b630eaceb9e0.slice/crio-8715b71944e9cbba828ecd43ade54d5dbc37f0e732f96ecbe3a6d266add543d8 WatchSource:0}: Error finding container 8715b71944e9cbba828ecd43ade54d5dbc37f0e732f96ecbe3a6d266add543d8: Status 404 returned error can't find the container with id 8715b71944e9cbba828ecd43ade54d5dbc37f0e732f96ecbe3a6d266add543d8
Sep 16 21:34:17.183433 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:17.183349    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:17.509452 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:17.509419    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl" event={"ID":"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0","Type":"ContainerStarted","Data":"8715b71944e9cbba828ecd43ade54d5dbc37f0e732f96ecbe3a6d266add543d8"}
Sep 16 21:34:18.010677 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:18.010649    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:18.165482 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:18.165453    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:18.517058 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:18.517015    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl" event={"ID":"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0","Type":"ContainerStarted","Data":"9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a"}
Sep 16 21:34:18.517235 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:18.517163    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:18.561084 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:18.561034    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl" podStartSLOduration=2.975015558 podStartE2EDuration="3.561016428s" totalImagesPullingTime="586.00087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:18.556453971 +0000 UTC m=+745.204865960" watchObservedRunningTime="2026-09-16 21:34:18.561016428 +0000 UTC m=+745.209428417"
Sep 16 21:34:18.588498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:18.588458    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:18.708003 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:18.707958    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:19.263462 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:19.263420    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:19.548448 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:19.548366    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:19.694737 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:19.694703    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:20.116737 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:20.116694    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:21.229856 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:21.229826    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:21.520679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:21.520636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:22.243827 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:22.243790    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:23.293880 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:23.293847    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:24.030353 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.030318    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:24.202802 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.202760    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:24.346993 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.346885    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:24.528027 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.527995    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:34:24.893567 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.893535    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"]
Sep 16 21:34:24.897325 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.897304    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:24.900412 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.900383    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-44e517-serving-cert\""
Sep 16 21:34:24.923598 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.923568    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"]
Sep 16 21:34:24.942911 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.942864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:24.998397 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.998368    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-maas-api-tls\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:24.998569 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.998433    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2v8\" (UniqueName: \"kubernetes.io/projected/081c381f-9818-4f06-831b-64ed0fa73fe8-kube-api-access-8t2v8\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:24.998569 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.998490    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081c381f-9818-4f06-831b-64ed0fa73fe8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:24.998569 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:24.998520    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-metrics-certs\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.041174 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.041144    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:25.099386 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.099338    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-maas-api-tls\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.099599 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.099417    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2v8\" (UniqueName: \"kubernetes.io/projected/081c381f-9818-4f06-831b-64ed0fa73fe8-kube-api-access-8t2v8\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.099599 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.099461    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081c381f-9818-4f06-831b-64ed0fa73fe8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.099599 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.099492    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-metrics-certs\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.100290 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.100259    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081c381f-9818-4f06-831b-64ed0fa73fe8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.101871 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.101846    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-metrics-certs\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.102054 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.102036    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-maas-api-tls\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.108699 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.108672    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2v8\" (UniqueName: \"kubernetes.io/projected/081c381f-9818-4f06-831b-64ed0fa73fe8-kube-api-access-8t2v8\") pod \"maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.207897 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.207807    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:25.493946 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.493881    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"]
Sep 16 21:34:25.496074 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:34:25.496042    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081c381f_9818_4f06_831b_64ed0fa73fe8.slice/crio-286686eaceba51866643669a116408d0da21c2d57f73c56afbaa439e5e6bae6e WatchSource:0}: Error finding container 286686eaceba51866643669a116408d0da21c2d57f73c56afbaa439e5e6bae6e: Status 404 returned error can't find the container with id 286686eaceba51866643669a116408d0da21c2d57f73c56afbaa439e5e6bae6e
Sep 16 21:34:25.521381 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.521341    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:25.554916 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.554877    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x" event={"ID":"081c381f-9818-4f06-831b-64ed0fa73fe8","Type":"ContainerStarted","Data":"286686eaceba51866643669a116408d0da21c2d57f73c56afbaa439e5e6bae6e"}
Sep 16 21:34:25.620965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:25.620891    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:26.085419 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:26.085320    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:26.561660 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:26.561626    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x" event={"ID":"081c381f-9818-4f06-831b-64ed0fa73fe8","Type":"ContainerStarted","Data":"b6869a9eab9572e576155c87e66d7170a3fb0552534c71c461eefaa540191e75"}
Sep 16 21:34:26.562091 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:26.561737    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:26.585315 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:26.585268    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x" podStartSLOduration=2.290381223 podStartE2EDuration="2.585250688s" totalImagesPullingTime="294.869465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:34:26.584119158 +0000 UTC m=+753.232531158" watchObservedRunningTime="2026-09-16 21:34:26.585250688 +0000 UTC m=+753.233662676"
Sep 16 21:34:27.459093 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:27.459061    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:28.582442 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:28.582412    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:29.497977 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:29.497946    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:30.491986 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:30.491936    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:31.112235 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:31.112197    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:31.220920 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:31.220867    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:31.377840 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:31.377738    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:31.579486 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:31.579444    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:31.681285 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:31.681151    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:32.317564 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:32.317524    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:32.571833 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:32.571753    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:34:33.229282 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:33.229249    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:34.102214 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:34.102176    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:35.012707 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:35.012671    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:35.721186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:35.721144    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:36.540116 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:36.540073    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:37.389429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:37.389366    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:37.746872 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:37.746779    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:37.862841 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:37.862799    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:37.958350 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:37.958319    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:38.116047 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:38.116016    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:38.844400 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:38.844352    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:46.012486 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:46.012457    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:34:59.824405 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:34:59.824366    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:00.358312 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.358271    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"]
Sep 16 21:35:00.358524 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.358481    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl" podUID="7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" containerName="maas-api" containerID="cri-o://9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a" gracePeriod=30
Sep 16 21:35:00.382475 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.382451    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:00.448628 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.448592    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:00.550052 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.550021    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:00.619271 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.619208    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:35:00.641391 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.641357    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:00.735477 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.735439    2567 generic.go:356] "Generic (PLEG): container finished" podID="7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" containerID="9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a" exitCode=0
Sep 16 21:35:00.735656 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.735503    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"
Sep 16 21:35:00.735656 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.735523    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl" event={"ID":"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0","Type":"ContainerDied","Data":"9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a"}
Sep 16 21:35:00.735656 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.735581    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl" event={"ID":"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0","Type":"ContainerDied","Data":"8715b71944e9cbba828ecd43ade54d5dbc37f0e732f96ecbe3a6d266add543d8"}
Sep 16 21:35:00.735656 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.735604    2567 scope.go:117] "RemoveContainer" containerID="9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a"
Sep 16 21:35:00.768313 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.766949    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-maas-api-tls\") pod \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") "
Sep 16 21:35:00.768313 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.767010    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-metrics-certs\") pod \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") "
Sep 16 21:35:00.768313 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.767050    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4bkd\" (UniqueName: \"kubernetes.io/projected/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-kube-api-access-p4bkd\") pod \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") "
Sep 16 21:35:00.768313 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.767202    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-openshift-service-ca\") pod \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\" (UID: \"7b467dc4-35a9-45f1-9bd2-b630eaceb9e0\") "
Sep 16 21:35:00.768313 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.767973    2567 scope.go:117] "RemoveContainer" containerID="9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a"
Sep 16 21:35:00.768724 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.768557    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:00.768724 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.768645    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" (UID: "7b467dc4-35a9-45f1-9bd2-b630eaceb9e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:35:00.769024 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:35:00.768930    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a\": container with ID starting with 9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a not found: ID does not exist" containerID="9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a"
Sep 16 21:35:00.769024 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.768986    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a"} err="failed to get container status \"9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a\": rpc error: code = NotFound desc = could not find container \"9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a\": container with ID starting with 9c98e173146d845f20e74944dc187a0500ab3ee5e2d5d306ddb4465bef5ab62a not found: ID does not exist"
Sep 16 21:35:00.771375 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.771350    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" (UID: "7b467dc4-35a9-45f1-9bd2-b630eaceb9e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:35:00.771485 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.771406    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" (UID: "7b467dc4-35a9-45f1-9bd2-b630eaceb9e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:35:00.772991 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.772970    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-kube-api-access-p4bkd" (OuterVolumeSpecName: "kube-api-access-p4bkd") pod "7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" (UID: "7b467dc4-35a9-45f1-9bd2-b630eaceb9e0"). InnerVolumeSpecName "kube-api-access-p4bkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:00.868217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.868179    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:35:00.868217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.868211    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:35:00.868217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.868219    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:35:00.868217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:00.868227    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bkd\" (UniqueName: \"kubernetes.io/projected/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0-kube-api-access-p4bkd\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:35:01.058549 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:01.058517    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"]
Sep 16 21:35:01.062599 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:01.062575    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e078853-996dfd9cc-zbkjl"]
Sep 16 21:35:01.071529 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:01.071502    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:01.217079 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:01.217039    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:01.313709 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:01.313606    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:01.428012 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:01.427965    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:01.733536 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:01.733445    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:01.920799 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:01.920758    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:01.942328 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:01.942296    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" path="/var/lib/kubelet/pods/7b467dc4-35a9-45f1-9bd2-b630eaceb9e0/volumes"
Sep 16 21:35:02.146663 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:02.146626    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:02.917304 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:02.917264    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:03.717839 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:03.717788    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:04.370584 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:04.370548    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:04.638948 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:04.638825    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:04.739338 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:04.739290    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:05.307826 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:05.307796    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:06.099693 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.099655    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"]
Sep 16 21:35:06.100172 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.100106    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" containerName="maas-api"
Sep 16 21:35:06.100172 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.100122    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" containerName="maas-api"
Sep 16 21:35:06.100285 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.100197    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b467dc4-35a9-45f1-9bd2-b630eaceb9e0" containerName="maas-api"
Sep 16 21:35:06.104830 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.104804    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.107951 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.107929    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-595d23dd-serving-cert\""
Sep 16 21:35:06.118117 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.118087    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d2e336-dcce-4e19-9566-5b632725e6f8-openshift-service-ca\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.118264 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.118131    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-metrics-certs\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.118264 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.118207    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8b5g\" (UniqueName: \"kubernetes.io/projected/37d2e336-dcce-4e19-9566-5b632725e6f8-kube-api-access-m8b5g\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.118395 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.118334    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-maas-api-tls\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.118552 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.118525    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"]
Sep 16 21:35:06.133233 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.133198    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:06.218778 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.218741    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8b5g\" (UniqueName: \"kubernetes.io/projected/37d2e336-dcce-4e19-9566-5b632725e6f8-kube-api-access-m8b5g\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.219032 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.218842    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-maas-api-tls\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.219032 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.218933    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d2e336-dcce-4e19-9566-5b632725e6f8-openshift-service-ca\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.219032 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.218963    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-metrics-certs\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.219032 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:35:06.218997    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-serving-cert: secret "maas-api-e2e-migrate-595d23dd-serving-cert" not found
Sep 16 21:35:06.219286 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:35:06.219085    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-maas-api-tls podName:37d2e336-dcce-4e19-9566-5b632725e6f8 nodeName:}" failed. No retries permitted until 2026-09-16 21:35:06.719064727 +0000 UTC m=+793.367476702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-maas-api-tls") pod "maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d" (UID: "37d2e336-dcce-4e19-9566-5b632725e6f8") : secret "maas-api-e2e-migrate-595d23dd-serving-cert" not found
Sep 16 21:35:06.219645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.219620    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d2e336-dcce-4e19-9566-5b632725e6f8-openshift-service-ca\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.221458 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.221432    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-metrics-certs\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.227617 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.227592    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8b5g\" (UniqueName: \"kubernetes.io/projected/37d2e336-dcce-4e19-9566-5b632725e6f8-kube-api-access-m8b5g\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.234131 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.234100    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:06.367300 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.367214    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:06.722381 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.722295    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-maas-api-tls\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.724759 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.724738    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-maas-api-tls\") pod \"maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:06.742147 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.742115    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:06.850176 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:06.850144    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:07.017889 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:07.017858    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:07.291082 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:07.291058    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"]
Sep 16 21:35:07.293423 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:35:07.293397    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d2e336_dcce_4e19_9566_5b632725e6f8.slice/crio-053c4756fb8c78378e93594691744a5d313960e093078cba5cb9d0343cc8cb2a WatchSource:0}: Error finding container 053c4756fb8c78378e93594691744a5d313960e093078cba5cb9d0343cc8cb2a: Status 404 returned error can't find the container with id 053c4756fb8c78378e93594691744a5d313960e093078cba5cb9d0343cc8cb2a
Sep 16 21:35:07.333929 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:07.333874    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:07.780132 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:07.780089    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d" event={"ID":"37d2e336-dcce-4e19-9566-5b632725e6f8","Type":"ContainerStarted","Data":"77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076"}
Sep 16 21:35:07.780298 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:07.780193    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d" event={"ID":"37d2e336-dcce-4e19-9566-5b632725e6f8","Type":"ContainerStarted","Data":"053c4756fb8c78378e93594691744a5d313960e093078cba5cb9d0343cc8cb2a"}
Sep 16 21:35:07.781776 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:07.781326    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:07.805638 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:07.805584    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d" podStartSLOduration=1.39780504 podStartE2EDuration="1.805567125s" totalImagesPullingTime="407.762085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:07.804199258 +0000 UTC m=+794.452611247" watchObservedRunningTime="2026-09-16 21:35:07.805567125 +0000 UTC m=+794.453979113"
Sep 16 21:35:08.193081 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:08.193044    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:09.233816 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:09.233778    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:10.026608 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:10.026568    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:10.824987 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:10.824953    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:14.795600 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:14.795563    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:19.084389 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:19.084314    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:28.451640 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:28.451597    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:29.341458 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:29.341419    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:30.069565 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:30.069526    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:30.851682 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:30.851647    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:31.646502 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:31.646468    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:32.358887 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:32.358846    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:33.223576 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:33.223547    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:34.021447 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:34.021406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:34.125424 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:34.125386    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:35.023604 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:35.023566    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:35.773038 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:35.773000    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:36.323301 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.323260    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"]
Sep 16 21:35:36.323757 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.323517    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d" podUID="37d2e336-dcce-4e19-9566-5b632725e6f8" containerName="maas-api" containerID="cri-o://77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076" gracePeriod=30
Sep 16 21:35:36.353200 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.353168    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:36.456407 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.456370    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:36.552293 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.552245    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:36.586470 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.586411    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:36.615683 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.615601    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:36.704022 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.703984    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8b5g\" (UniqueName: \"kubernetes.io/projected/37d2e336-dcce-4e19-9566-5b632725e6f8-kube-api-access-m8b5g\") pod \"37d2e336-dcce-4e19-9566-5b632725e6f8\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") "
Sep 16 21:35:36.704220 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.704058    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-maas-api-tls\") pod \"37d2e336-dcce-4e19-9566-5b632725e6f8\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") "
Sep 16 21:35:36.704220 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.704102    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d2e336-dcce-4e19-9566-5b632725e6f8-openshift-service-ca\") pod \"37d2e336-dcce-4e19-9566-5b632725e6f8\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") "
Sep 16 21:35:36.704220 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.704184    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-metrics-certs\") pod \"37d2e336-dcce-4e19-9566-5b632725e6f8\" (UID: \"37d2e336-dcce-4e19-9566-5b632725e6f8\") "
Sep 16 21:35:36.704483 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.704458    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d2e336-dcce-4e19-9566-5b632725e6f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37d2e336-dcce-4e19-9566-5b632725e6f8" (UID: "37d2e336-dcce-4e19-9566-5b632725e6f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:35:36.706408 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.706295    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:36.706717 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.706682    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d2e336-dcce-4e19-9566-5b632725e6f8-kube-api-access-m8b5g" (OuterVolumeSpecName: "kube-api-access-m8b5g") pod "37d2e336-dcce-4e19-9566-5b632725e6f8" (UID: "37d2e336-dcce-4e19-9566-5b632725e6f8"). InnerVolumeSpecName "kube-api-access-m8b5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:35:36.706812 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.706740    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "37d2e336-dcce-4e19-9566-5b632725e6f8" (UID: "37d2e336-dcce-4e19-9566-5b632725e6f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:35:36.707001 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.706982    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37d2e336-dcce-4e19-9566-5b632725e6f8" (UID: "37d2e336-dcce-4e19-9566-5b632725e6f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:35:36.805922 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.805873    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:35:36.806150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.805950    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8b5g\" (UniqueName: \"kubernetes.io/projected/37d2e336-dcce-4e19-9566-5b632725e6f8-kube-api-access-m8b5g\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:35:36.806150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.805966    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37d2e336-dcce-4e19-9566-5b632725e6f8-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:35:36.806150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.805978    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d2e336-dcce-4e19-9566-5b632725e6f8-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:35:36.931268 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.931166    2567 generic.go:356] "Generic (PLEG): container finished" podID="37d2e336-dcce-4e19-9566-5b632725e6f8" containerID="77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076" exitCode=0
Sep 16 21:35:36.931268 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.931213    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"
Sep 16 21:35:36.931525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.931270    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d" event={"ID":"37d2e336-dcce-4e19-9566-5b632725e6f8","Type":"ContainerDied","Data":"77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076"}
Sep 16 21:35:36.931525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.931334    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d" event={"ID":"37d2e336-dcce-4e19-9566-5b632725e6f8","Type":"ContainerDied","Data":"053c4756fb8c78378e93594691744a5d313960e093078cba5cb9d0343cc8cb2a"}
Sep 16 21:35:36.931525 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.931363    2567 scope.go:117] "RemoveContainer" containerID="77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076"
Sep 16 21:35:36.949988 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.949960    2567 scope.go:117] "RemoveContainer" containerID="77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076"
Sep 16 21:35:36.950344 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:35:36.950323    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076\": container with ID starting with 77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076 not found: ID does not exist" containerID="77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076"
Sep 16 21:35:36.950465 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.950357    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076"} err="failed to get container status \"77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076\": rpc error: code = NotFound desc = could not find container \"77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076\": container with ID starting with 77a8d5e882e843da5de9db2142229a16008971191f4be30ab45250e4e6cb1076 not found: ID does not exist"
Sep 16 21:35:36.958272 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.958247    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"]
Sep 16 21:35:36.961866 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:36.961845    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-595d23dd-66f6d4d6b8-k6w8d"]
Sep 16 21:35:37.005695 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:37.005653    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:37.130436 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:37.130399    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:37.449571 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:37.449533    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:37.558006 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:37.557957    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:37.942877 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:37.942831    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d2e336-dcce-4e19-9566-5b632725e6f8" path="/var/lib/kubelet/pods/37d2e336-dcce-4e19-9566-5b632725e6f8/volumes"
Sep 16 21:35:37.987829 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:37.987791    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:38.714033 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:38.713996    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:39.525534 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:39.525501    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:39.738213 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:39.738177    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:39.832217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:39.832118    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:40.558781 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:40.558744    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:41.263385 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:41.263357    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:42.105839 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.105795    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"]
Sep 16 21:35:42.106528 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.106502    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d2e336-dcce-4e19-9566-5b632725e6f8" containerName="maas-api"
Sep 16 21:35:42.106528 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.106529    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d2e336-dcce-4e19-9566-5b632725e6f8" containerName="maas-api"
Sep 16 21:35:42.106680 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.106668    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d2e336-dcce-4e19-9566-5b632725e6f8" containerName="maas-api"
Sep 16 21:35:42.111378 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.111353    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.113803 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.113779    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8eafdc51-serving-cert\""
Sep 16 21:35:42.121107 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.121079    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"]
Sep 16 21:35:42.135466 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.135432    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:42.237957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.237921    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:42.259103 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.259066    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-metrics-certs\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.259254 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.259112    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e49539-71c0-4099-b168-9e4aa2d82639-openshift-service-ca\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.259254 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.259168    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-maas-api-tls\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.259347 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.259286    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txbvk\" (UniqueName: \"kubernetes.io/projected/42e49539-71c0-4099-b168-9e4aa2d82639-kube-api-access-txbvk\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.336214 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.336176    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:42.359884 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.359801    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txbvk\" (UniqueName: \"kubernetes.io/projected/42e49539-71c0-4099-b168-9e4aa2d82639-kube-api-access-txbvk\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.359884 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.359863    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-metrics-certs\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.360068 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.359890    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e49539-71c0-4099-b168-9e4aa2d82639-openshift-service-ca\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.360068 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.359944    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-maas-api-tls\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.360624 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.360598    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e49539-71c0-4099-b168-9e4aa2d82639-openshift-service-ca\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.362416 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.362385    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-metrics-certs\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.362568 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.362539    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-maas-api-tls\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.368852 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.368834    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txbvk\" (UniqueName: \"kubernetes.io/projected/42e49539-71c0-4099-b168-9e4aa2d82639-kube-api-access-txbvk\") pod \"maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.425644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.425605    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:42.697777 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.697736    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"]
Sep 16 21:35:42.700429 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:35:42.700384    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e49539_71c0_4099_b168_9e4aa2d82639.slice/crio-b595f2eff402778e61509fe5e119b5b46cc66445f9b337f7cefb600e1d8e44b1 WatchSource:0}: Error finding container b595f2eff402778e61509fe5e119b5b46cc66445f9b337f7cefb600e1d8e44b1: Status 404 returned error can't find the container with id b595f2eff402778e61509fe5e119b5b46cc66445f9b337f7cefb600e1d8e44b1
Sep 16 21:35:42.779676 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.779642    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:42.853393 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.853342    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:42.962633 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:42.962557    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr" event={"ID":"42e49539-71c0-4099-b168-9e4aa2d82639","Type":"ContainerStarted","Data":"b595f2eff402778e61509fe5e119b5b46cc66445f9b337f7cefb600e1d8e44b1"}
Sep 16 21:35:43.134419 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:43.134379    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:43.935412 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:43.935364    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:43.968964 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:43.968920    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr" event={"ID":"42e49539-71c0-4099-b168-9e4aa2d82639","Type":"ContainerStarted","Data":"ee718f27dfcd57dacd15090243e3ac7e7f0d9dbf2d05d347e68ffaeb3db9ac0c"}
Sep 16 21:35:43.969129 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:43.969038    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:35:43.985848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:43.985785    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr" podStartSLOduration=1.6710236680000001 podStartE2EDuration="1.985768544s" totalImagesPullingTime="314.744876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:35:43.984136162 +0000 UTC m=+830.632548150" watchObservedRunningTime="2026-09-16 21:35:43.985768544 +0000 UTC m=+830.634180531"
Sep 16 21:35:44.616344 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:44.616305    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:45.266847 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:45.266818    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:47.889691 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:47.889658    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:35:49.979437 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:35:49.979412    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:36:02.309676 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:02.309643    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:04.214980 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:04.214945    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:04.851960 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:04.851916    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:05.716836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:05.716799    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:06.646096 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:06.646056    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:07.490807 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:07.490772    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:08.113877 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:08.113839    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:08.947132 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:08.947095    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:09.633887 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:09.633837    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:10.604888 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:10.604846    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:11.416708 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:11.416673    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:11.971858 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:11.971825    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"]
Sep 16 21:36:11.972273 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:11.972068    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr" podUID="42e49539-71c0-4099-b168-9e4aa2d82639" containerName="maas-api" containerID="cri-o://ee718f27dfcd57dacd15090243e3ac7e7f0d9dbf2d05d347e68ffaeb3db9ac0c" gracePeriod=30
Sep 16 21:36:12.000640 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.000601    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:12.076878 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.076831    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:12.112366 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.112335    2567 generic.go:356] "Generic (PLEG): container finished" podID="42e49539-71c0-4099-b168-9e4aa2d82639" containerID="ee718f27dfcd57dacd15090243e3ac7e7f0d9dbf2d05d347e68ffaeb3db9ac0c" exitCode=0
Sep 16 21:36:12.112539 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.112417    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr" event={"ID":"42e49539-71c0-4099-b168-9e4aa2d82639","Type":"ContainerDied","Data":"ee718f27dfcd57dacd15090243e3ac7e7f0d9dbf2d05d347e68ffaeb3db9ac0c"}
Sep 16 21:36:12.177766 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.177717    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:12.232645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.232591    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:36:12.257204 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.257167    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:12.350048 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.350018    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-metrics-certs\") pod \"42e49539-71c0-4099-b168-9e4aa2d82639\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") "
Sep 16 21:36:12.350248 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.350133    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-maas-api-tls\") pod \"42e49539-71c0-4099-b168-9e4aa2d82639\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") "
Sep 16 21:36:12.350248 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.350215    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txbvk\" (UniqueName: \"kubernetes.io/projected/42e49539-71c0-4099-b168-9e4aa2d82639-kube-api-access-txbvk\") pod \"42e49539-71c0-4099-b168-9e4aa2d82639\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") "
Sep 16 21:36:12.350374 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.350271    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e49539-71c0-4099-b168-9e4aa2d82639-openshift-service-ca\") pod \"42e49539-71c0-4099-b168-9e4aa2d82639\" (UID: \"42e49539-71c0-4099-b168-9e4aa2d82639\") "
Sep 16 21:36:12.350949 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.350888    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e49539-71c0-4099-b168-9e4aa2d82639-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42e49539-71c0-4099-b168-9e4aa2d82639" (UID: "42e49539-71c0-4099-b168-9e4aa2d82639"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:12.352423 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.352363    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "42e49539-71c0-4099-b168-9e4aa2d82639" (UID: "42e49539-71c0-4099-b168-9e4aa2d82639"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:36:12.352689 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.352668    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42e49539-71c0-4099-b168-9e4aa2d82639" (UID: "42e49539-71c0-4099-b168-9e4aa2d82639"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:36:12.352821 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.352804    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e49539-71c0-4099-b168-9e4aa2d82639-kube-api-access-txbvk" (OuterVolumeSpecName: "kube-api-access-txbvk") pod "42e49539-71c0-4099-b168-9e4aa2d82639" (UID: "42e49539-71c0-4099-b168-9e4aa2d82639"). InnerVolumeSpecName "kube-api-access-txbvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:12.451174 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.451136    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txbvk\" (UniqueName: \"kubernetes.io/projected/42e49539-71c0-4099-b168-9e4aa2d82639-kube-api-access-txbvk\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:12.451174 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.451164    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e49539-71c0-4099-b168-9e4aa2d82639-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:12.451174 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.451174    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:12.451174 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.451183    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e49539-71c0-4099-b168-9e4aa2d82639-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:12.684072 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:12.684021    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:13.119442 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:13.119405    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr" event={"ID":"42e49539-71c0-4099-b168-9e4aa2d82639","Type":"ContainerDied","Data":"b595f2eff402778e61509fe5e119b5b46cc66445f9b337f7cefb600e1d8e44b1"}
Sep 16 21:36:13.119880 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:13.119463    2567 scope.go:117] "RemoveContainer" containerID="ee718f27dfcd57dacd15090243e3ac7e7f0d9dbf2d05d347e68ffaeb3db9ac0c"
Sep 16 21:36:13.119880 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:13.119415    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"
Sep 16 21:36:13.154456 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:13.154424    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"]
Sep 16 21:36:13.157592 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:13.157563    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8eafdc51-65cbd79b6b-nhdxr"]
Sep 16 21:36:13.338100 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:13.338066    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:13.747144 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:13.747107    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:13.854955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:13.854919    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:13.942324 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:13.942283    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e49539-71c0-4099-b168-9e4aa2d82639" path="/var/lib/kubelet/pods/42e49539-71c0-4099-b168-9e4aa2d82639/volumes"
Sep 16 21:36:14.280593 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:14.280561    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:14.443757 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:14.443723    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:15.025364 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:15.025318    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:15.718926 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:15.718867    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:16.475378 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:16.475343    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:16.997501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:16.997468    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:17.102274 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:17.102242    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:17.452698 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:17.452605    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:18.141447 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.141406    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"]
Sep 16 21:36:18.142118 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.142097    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e49539-71c0-4099-b168-9e4aa2d82639" containerName="maas-api"
Sep 16 21:36:18.142205 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.142121    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e49539-71c0-4099-b168-9e4aa2d82639" containerName="maas-api"
Sep 16 21:36:18.142263 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.142251    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e49539-71c0-4099-b168-9e4aa2d82639" containerName="maas-api"
Sep 16 21:36:18.147250 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.147229    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.149925 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.149873    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-a1fb6d89-serving-cert\""
Sep 16 21:36:18.152803 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.152767    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"]
Sep 16 21:36:18.172320 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.172287    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:18.273163 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.273130    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:18.309277 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.309244    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-openshift-service-ca\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.309432 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.309329    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-maas-api-tls\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.309432 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.309374    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-metrics-certs\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.309432 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.309409    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgm6\" (UniqueName: \"kubernetes.io/projected/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-kube-api-access-xhgm6\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.410312 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.410235    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-openshift-service-ca\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.410312 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.410301    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-maas-api-tls\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.410499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.410323    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-metrics-certs\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.410499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.410349    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgm6\" (UniqueName: \"kubernetes.io/projected/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-kube-api-access-xhgm6\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.410499 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:36:18.410450    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-serving-cert: secret "maas-api-e2e-derive-a1fb6d89-serving-cert" not found
Sep 16 21:36:18.410617 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:36:18.410543    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-maas-api-tls podName:c42faba1-ffb0-4602-b2ca-aa65c898f8cd nodeName:}" failed. No retries permitted until 2026-09-16 21:36:18.910519417 +0000 UTC m=+865.558931598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-maas-api-tls") pod "maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5" (UID: "c42faba1-ffb0-4602-b2ca-aa65c898f8cd") : secret "maas-api-e2e-derive-a1fb6d89-serving-cert" not found
Sep 16 21:36:18.411003 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.410984    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-openshift-service-ca\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.413149 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.413122    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-metrics-certs\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.418468 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.418442    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgm6\" (UniqueName: \"kubernetes.io/projected/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-kube-api-access-xhgm6\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.554667 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.554630    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:18.687654 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.687566    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:18.818427 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.818398    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:18.916333 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.916302    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-maas-api-tls\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:18.918831 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:18.918806    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-maas-api-tls\") pod \"maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:19.063224 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:19.063182    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:19.148423 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:19.148377    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:19.335637 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:19.335612    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"]
Sep 16 21:36:19.337487 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:36:19.337460    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42faba1_ffb0_4602_b2ca_aa65c898f8cd.slice/crio-52ad9ea2e785c60bab0285ec404da148a38f6faeb9bbb9babeb1be57bc9269c8 WatchSource:0}: Error finding container 52ad9ea2e785c60bab0285ec404da148a38f6faeb9bbb9babeb1be57bc9269c8: Status 404 returned error can't find the container with id 52ad9ea2e785c60bab0285ec404da148a38f6faeb9bbb9babeb1be57bc9269c8
Sep 16 21:36:20.053663 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:20.053626    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:20.159254 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:20.159223    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5" event={"ID":"c42faba1-ffb0-4602-b2ca-aa65c898f8cd","Type":"ContainerStarted","Data":"52ad9ea2e785c60bab0285ec404da148a38f6faeb9bbb9babeb1be57bc9269c8"}
Sep 16 21:36:20.819241 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:20.819191    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:21.166646 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:21.166561    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5" event={"ID":"c42faba1-ffb0-4602-b2ca-aa65c898f8cd","Type":"ContainerStarted","Data":"370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810"}
Sep 16 21:36:21.167026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:21.166708    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:21.187472 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:21.187422    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5" podStartSLOduration=2.398053352 podStartE2EDuration="3.187406398s" totalImagesPullingTime="789.353046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:21.186086769 +0000 UTC m=+867.834498760" watchObservedRunningTime="2026-09-16 21:36:21.187406398 +0000 UTC m=+867.835818386"
Sep 16 21:36:21.638384 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:21.638348    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:21.883354 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:21.883323    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:22.466492 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:22.466462    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:23.166649 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:23.166604    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:23.939369 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:23.939330    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:24.697645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:24.697604    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:25.421219 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:25.421180    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:27.176893 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:27.176860    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:30.240538 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:30.240503    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:30.817939 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:30.817886    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"]
Sep 16 21:36:30.818181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:30.818131    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52" podUID="dc7fef8b-dc67-42c0-a80d-d381375dc03f" containerName="maas-api" containerID="cri-o://90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b" gracePeriod=30
Sep 16 21:36:30.885189 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:30.885140    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:31.029196 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.029159    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:31.097727 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.097702    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:36:31.126607 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.126569    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:31.134558 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.134529    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fef8b-dc67-42c0-a80d-d381375dc03f-openshift-service-ca\") pod \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") "
Sep 16 21:36:31.134724 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.134599    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mg4z\" (UniqueName: \"kubernetes.io/projected/dc7fef8b-dc67-42c0-a80d-d381375dc03f-kube-api-access-7mg4z\") pod \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") "
Sep 16 21:36:31.134724 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.134645    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-maas-api-tls\") pod \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") "
Sep 16 21:36:31.134831 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.134738    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-metrics-certs\") pod \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\" (UID: \"dc7fef8b-dc67-42c0-a80d-d381375dc03f\") "
Sep 16 21:36:31.134982 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.134959    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7fef8b-dc67-42c0-a80d-d381375dc03f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc7fef8b-dc67-42c0-a80d-d381375dc03f" (UID: "dc7fef8b-dc67-42c0-a80d-d381375dc03f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:31.135096 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.135077    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc7fef8b-dc67-42c0-a80d-d381375dc03f-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:31.136826 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.136795    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc7fef8b-dc67-42c0-a80d-d381375dc03f" (UID: "dc7fef8b-dc67-42c0-a80d-d381375dc03f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:36:31.136953 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.136866    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc7fef8b-dc67-42c0-a80d-d381375dc03f" (UID: "dc7fef8b-dc67-42c0-a80d-d381375dc03f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:36:31.136953 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.136884    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7fef8b-dc67-42c0-a80d-d381375dc03f-kube-api-access-7mg4z" (OuterVolumeSpecName: "kube-api-access-7mg4z") pod "dc7fef8b-dc67-42c0-a80d-d381375dc03f" (UID: "dc7fef8b-dc67-42c0-a80d-d381375dc03f"). InnerVolumeSpecName "kube-api-access-7mg4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:31.190552 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.190513    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:31.222176 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.222144    2567 generic.go:356] "Generic (PLEG): container finished" podID="dc7fef8b-dc67-42c0-a80d-d381375dc03f" containerID="90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b" exitCode=0
Sep 16 21:36:31.222357 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.222187    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"
Sep 16 21:36:31.222357 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.222190    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52" event={"ID":"dc7fef8b-dc67-42c0-a80d-d381375dc03f","Type":"ContainerDied","Data":"90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b"}
Sep 16 21:36:31.222357 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.222290    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52" event={"ID":"dc7fef8b-dc67-42c0-a80d-d381375dc03f","Type":"ContainerDied","Data":"0e38c12066f6325a15d97af1011f1391a9b532bcdaf89208850203996ab245cb"}
Sep 16 21:36:31.222357 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.222313    2567 scope.go:117] "RemoveContainer" containerID="90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b"
Sep 16 21:36:31.232190 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.232169    2567 scope.go:117] "RemoveContainer" containerID="90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b"
Sep 16 21:36:31.232430 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:36:31.232413    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b\": container with ID starting with 90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b not found: ID does not exist" containerID="90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b"
Sep 16 21:36:31.232481 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.232437    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b"} err="failed to get container status \"90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b\": rpc error: code = NotFound desc = could not find container \"90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b\": container with ID starting with 90ab3a5079273dc0826b461a441e5be9f669e9ad45fe6ddda9800bb12ab51c0b not found: ID does not exist"
Sep 16 21:36:31.236464 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.236446    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mg4z\" (UniqueName: \"kubernetes.io/projected/dc7fef8b-dc67-42c0-a80d-d381375dc03f-kube-api-access-7mg4z\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:31.236541 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.236466    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:31.236541 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.236475    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc7fef8b-dc67-42c0-a80d-d381375dc03f-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:31.245182 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.245157    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"]
Sep 16 21:36:31.249456 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.249433    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-ddbefa-76555c5c8-nxk52"]
Sep 16 21:36:31.496449 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.496355    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:31.596569 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.596533    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:31.942880 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:31.942847    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc7fef8b-dc67-42c0-a80d-d381375dc03f" path="/var/lib/kubelet/pods/dc7fef8b-dc67-42c0-a80d-d381375dc03f/volumes"
Sep 16 21:36:32.202024 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:32.201914    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:32.906224 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:32.906191    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:33.663719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:33.663682    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:34.537483 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:34.537428    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:35.334300 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:35.334262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:35.998234 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:35.998194    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:36.620834 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:36.620796    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:36.915929 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:36.915815    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:37.803561 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:37.803526    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:38.647505 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:38.647470    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:39.443102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:39.443064    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:40.250965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:40.250930    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:40.973791 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:40.973755    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:41.664406 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:41.664375    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:42.445494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:42.445449    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:42.957230 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:42.957198    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:43.186954 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:43.186912    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:43.649592 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:43.649561    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:43.959606 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:43.959503    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:44.096974 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:44.096937    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:46.320330 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:46.320248    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:46.636782 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:46.636705    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:46.724048 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:46.724010    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:47.230454 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:47.230419    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:48.098895 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:48.098850    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:48.893752 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:48.893717    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:49.080138 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:49.080105    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:49.933467 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:49.933431    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:50.709030 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.708990    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"]
Sep 16 21:36:50.709528 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.709509    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc7fef8b-dc67-42c0-a80d-d381375dc03f" containerName="maas-api"
Sep 16 21:36:50.709528 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.709530    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc7fef8b-dc67-42c0-a80d-d381375dc03f" containerName="maas-api"
Sep 16 21:36:50.709666 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.709641    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc7fef8b-dc67-42c0-a80d-d381375dc03f" containerName="maas-api"
Sep 16 21:36:50.712580 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.712561    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.714992 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.714972    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b91829-serving-cert\""
Sep 16 21:36:50.724015 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.723993    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"]
Sep 16 21:36:50.744706 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.744669    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:50.841706 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.841671    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-maas-api-tls\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.841896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.841732    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-metrics-certs\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.841896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.841802    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twx5k\" (UniqueName: \"kubernetes.io/projected/bf908481-41c0-40b5-b5a9-7c3e885f83c7-kube-api-access-twx5k\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.841896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.841836    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf908481-41c0-40b5-b5a9-7c3e885f83c7-openshift-service-ca\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.845669 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.845630    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:50.903275 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.903237    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:50.943375 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.943339    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-maas-api-tls\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.943866 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.943398    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-metrics-certs\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.943866 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.943422    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twx5k\" (UniqueName: \"kubernetes.io/projected/bf908481-41c0-40b5-b5a9-7c3e885f83c7-kube-api-access-twx5k\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.943866 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.943447    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf908481-41c0-40b5-b5a9-7c3e885f83c7-openshift-service-ca\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.944203 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.944184    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf908481-41c0-40b5-b5a9-7c3e885f83c7-openshift-service-ca\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.945871 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.945852    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-metrics-certs\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.945991 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.945923    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-maas-api-tls\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:50.952054 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:50.952026    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twx5k\" (UniqueName: \"kubernetes.io/projected/bf908481-41c0-40b5-b5a9-7c3e885f83c7-kube-api-access-twx5k\") pod \"maas-api-e2e-explicit-b91829-548798795d-xvrzb\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:51.025100 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.025067    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:51.226314 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.226278    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:51.315580 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.315335    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"]
Sep 16 21:36:51.317018 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.316981    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:51.318761 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:36:51.318731    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf908481_41c0_40b5_b5a9_7c3e885f83c7.slice/crio-ebf6d3bafc754bb4b3a2a1f5777ae5380114e381230c1da58dcb9ecc3eb61c2f WatchSource:0}: Error finding container ebf6d3bafc754bb4b3a2a1f5777ae5380114e381230c1da58dcb9ecc3eb61c2f: Status 404 returned error can't find the container with id ebf6d3bafc754bb4b3a2a1f5777ae5380114e381230c1da58dcb9ecc3eb61c2f
Sep 16 21:36:51.331447 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.331421    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb" event={"ID":"bf908481-41c0-40b5-b5a9-7c3e885f83c7","Type":"ContainerStarted","Data":"ebf6d3bafc754bb4b3a2a1f5777ae5380114e381230c1da58dcb9ecc3eb61c2f"}
Sep 16 21:36:51.498322 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.498244    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"]
Sep 16 21:36:51.498699 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.498519    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5" podUID="c42faba1-ffb0-4602-b2ca-aa65c898f8cd" containerName="maas-api" containerID="cri-o://370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810" gracePeriod=30
Sep 16 21:36:51.522877 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.522848    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:51.624454 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.624421    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:51.723454 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.723412    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:51.742503 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.742473    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:51.785799 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.785771    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:51.853191 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.853151    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-metrics-certs\") pod \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") "
Sep 16 21:36:51.853392 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.853241    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-openshift-service-ca\") pod \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") "
Sep 16 21:36:51.853392 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.853316    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhgm6\" (UniqueName: \"kubernetes.io/projected/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-kube-api-access-xhgm6\") pod \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") "
Sep 16 21:36:51.853392 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.853381    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-maas-api-tls\") pod \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\" (UID: \"c42faba1-ffb0-4602-b2ca-aa65c898f8cd\") "
Sep 16 21:36:51.853699 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.853665    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c42faba1-ffb0-4602-b2ca-aa65c898f8cd" (UID: "c42faba1-ffb0-4602-b2ca-aa65c898f8cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:36:51.855381 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.855348    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c42faba1-ffb0-4602-b2ca-aa65c898f8cd" (UID: "c42faba1-ffb0-4602-b2ca-aa65c898f8cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:36:51.855785 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.855763    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-kube-api-access-xhgm6" (OuterVolumeSpecName: "kube-api-access-xhgm6") pod "c42faba1-ffb0-4602-b2ca-aa65c898f8cd" (UID: "c42faba1-ffb0-4602-b2ca-aa65c898f8cd"). InnerVolumeSpecName "kube-api-access-xhgm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:36:51.855943 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.855895    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c42faba1-ffb0-4602-b2ca-aa65c898f8cd" (UID: "c42faba1-ffb0-4602-b2ca-aa65c898f8cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:36:51.922363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.922262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:51.955115 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.955004    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:51.955115 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.955032    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:51.955115 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.955045    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:51.955115 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:51.955061    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhgm6\" (UniqueName: \"kubernetes.io/projected/c42faba1-ffb0-4602-b2ca-aa65c898f8cd-kube-api-access-xhgm6\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:36:52.338628 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.338596    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb" event={"ID":"bf908481-41c0-40b5-b5a9-7c3e885f83c7","Type":"ContainerStarted","Data":"dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc"}
Sep 16 21:36:52.338819 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.338675    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:36:52.340057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.340033    2567 generic.go:356] "Generic (PLEG): container finished" podID="c42faba1-ffb0-4602-b2ca-aa65c898f8cd" containerID="370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810" exitCode=0
Sep 16 21:36:52.340205 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.340070    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"
Sep 16 21:36:52.340205 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.340109    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5" event={"ID":"c42faba1-ffb0-4602-b2ca-aa65c898f8cd","Type":"ContainerDied","Data":"370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810"}
Sep 16 21:36:52.340205 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.340152    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5" event={"ID":"c42faba1-ffb0-4602-b2ca-aa65c898f8cd","Type":"ContainerDied","Data":"52ad9ea2e785c60bab0285ec404da148a38f6faeb9bbb9babeb1be57bc9269c8"}
Sep 16 21:36:52.340205 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.340178    2567 scope.go:117] "RemoveContainer" containerID="370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810"
Sep 16 21:36:52.349572 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.349555    2567 scope.go:117] "RemoveContainer" containerID="370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810"
Sep 16 21:36:52.349841 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:36:52.349817    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810\": container with ID starting with 370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810 not found: ID does not exist" containerID="370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810"
Sep 16 21:36:52.349937 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.349851    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810"} err="failed to get container status \"370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810\": rpc error: code = NotFound desc = could not find container \"370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810\": container with ID starting with 370ff22805bacc84ec2170a4db2d6d18156f1038a1f9721cf3e69f4d0f916810 not found: ID does not exist"
Sep 16 21:36:52.364338 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.356091    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb" podStartSLOduration=1.898541869 podStartE2EDuration="2.356072797s" totalImagesPullingTime="457.530928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:36:52.3541508 +0000 UTC m=+899.002562789" watchObservedRunningTime="2026-09-16 21:36:52.356072797 +0000 UTC m=+899.004484787"
Sep 16 21:36:52.375024 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.374989    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"]
Sep 16 21:36:52.377026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.377002    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-a1fb6d89-565c8b9479-llqw5"]
Sep 16 21:36:52.592822 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.592732    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:52.691877 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.691841    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:52.790043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:52.790007    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:53.506640 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:53.506603    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:53.835540 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:53.835442    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:36:53.839057 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:53.839034    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:36:53.943193 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:53.943150    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c42faba1-ffb0-4602-b2ca-aa65c898f8cd" path="/var/lib/kubelet/pods/c42faba1-ffb0-4602-b2ca-aa65c898f8cd/volumes"
Sep 16 21:36:54.165328 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:54.165222    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:54.401295 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:54.401258    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:55.038074 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:55.038033    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:55.708314 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:55.708281    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:56.445760 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:56.445714    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:36:58.353116 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:36:58.353082    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:37:00.228878 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:00.228842    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:00.331239 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:00.331208    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:00.431187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:00.431148    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:00.529924 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:00.529867    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:00.912187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:00.912095    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:09.275795 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.275762    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"]
Sep 16 21:37:09.278118 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.276245    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42faba1-ffb0-4602-b2ca-aa65c898f8cd" containerName="maas-api"
Sep 16 21:37:09.278118 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.276259    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42faba1-ffb0-4602-b2ca-aa65c898f8cd" containerName="maas-api"
Sep 16 21:37:09.278118 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.276326    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42faba1-ffb0-4602-b2ca-aa65c898f8cd" containerName="maas-api"
Sep 16 21:37:09.279286 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.279267    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.282796 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.282766    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"openshift-service-ca.crt\""
Sep 16 21:37:09.282965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.282814    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"premium-w0-35f5fd-kserve-self-signed-certs\""
Sep 16 21:37:09.282965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.282832    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"kube-root-ca.crt\""
Sep 16 21:37:09.282965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.282848    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"default-dockercfg-94lx6\""
Sep 16 21:37:09.289150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.289128    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"]
Sep 16 21:37:09.294110 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.294083    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:09.315327 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.315298    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tls-certs\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.315469 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.315340    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5qd6\" (UniqueName: \"kubernetes.io/projected/b4a426c3-1bd3-4311-8bab-a9a419e77e16-kube-api-access-g5qd6\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.315469 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.315430    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-dshm\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.315469 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.315456    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-model-cache\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.315583 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.315520    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-home\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.315583 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.315566    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tmp-dir\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.416755 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.416712    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tls-certs\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.416989 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.416761    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5qd6\" (UniqueName: \"kubernetes.io/projected/b4a426c3-1bd3-4311-8bab-a9a419e77e16-kube-api-access-g5qd6\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.416989 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.416796    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-dshm\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.416989 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.416823    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-model-cache\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.416989 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.416969    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-home\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.417228 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.417039    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tmp-dir\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.417371 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.417352    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-model-cache\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.417446 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.417433    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-home\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.417493 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.417471    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tmp-dir\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.419130 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.419109    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-dshm\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.419282 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.419266    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tls-certs\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.424021 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.423999    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5qd6\" (UniqueName: \"kubernetes.io/projected/b4a426c3-1bd3-4311-8bab-a9a419e77e16-kube-api-access-g5qd6\") pod \"premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") " pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.589392 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.589291    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:09.860047 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:09.859977    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"]
Sep 16 21:37:09.862338 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:37:09.862313    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4a426c3_1bd3_4311_8bab_a9a419e77e16.slice/crio-0238b6dc6c99e05e19942961cd36b6c8e0fe7acf8cd905405940ee3decfb6ef3 WatchSource:0}: Error finding container 0238b6dc6c99e05e19942961cd36b6c8e0fe7acf8cd905405940ee3decfb6ef3: Status 404 returned error can't find the container with id 0238b6dc6c99e05e19942961cd36b6c8e0fe7acf8cd905405940ee3decfb6ef3
Sep 16 21:37:10.381859 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:10.381818    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:10.437741 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:10.437703    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm" event={"ID":"b4a426c3-1bd3-4311-8bab-a9a419e77e16","Type":"ContainerStarted","Data":"a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669"}
Sep 16 21:37:10.437741 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:10.437744    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm" event={"ID":"b4a426c3-1bd3-4311-8bab-a9a419e77e16","Type":"ContainerStarted","Data":"0238b6dc6c99e05e19942961cd36b6c8e0fe7acf8cd905405940ee3decfb6ef3"}
Sep 16 21:37:10.455742 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:10.455686    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm" podStartSLOduration=1.455670212 podStartE2EDuration="1.455670212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:37:10.453351388 +0000 UTC m=+917.101763378" watchObservedRunningTime="2026-09-16 21:37:10.455670212 +0000 UTC m=+917.104082234"
Sep 16 21:37:12.752594 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:12.752550    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:13.487330 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:13.487287    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:14.142675 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:14.142636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:14.926332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:14.926294    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:15.574312 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:15.574280    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:16.295343 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:16.295300    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:16.646887 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:16.646798    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:19.589683 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:19.589643    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:19.589683 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:19.589691    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:19.602373 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:19.602346    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:20.498917 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:20.498872    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:37:26.782999 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:26.782958    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:27.005807 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:27.005774    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:32.537387 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:32.537340    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:32.638537 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:32.638502    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:32.738049 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:32.738019    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:48.557863 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:48.557796    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:51.615241 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:51.615191    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:54.355358 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:54.355318    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:54.457366 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:54.457328    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:54.555738 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:54.555703    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:56.055439 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:56.055398    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:57.131803 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:57.131771    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:57.988277 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:57.988224    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:58.991835 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:58.991803    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:37:59.833048 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:37:59.833016    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:00.482492 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:00.482462    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:01.234800 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:01.234765    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:05.040975 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:05.040936    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:05.803774 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:05.803742    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:06.671212 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:06.671175    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"]
Sep 16 21:38:06.671686 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:06.671426    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb" podUID="bf908481-41c0-40b5-b5a9-7c3e885f83c7" containerName="maas-api" containerID="cri-o://dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc" gracePeriod=30
Sep 16 21:38:06.698798 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:06.698749    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:06.767652 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:06.767613    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:06.833978 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:06.833920    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:06.898214 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:06.898176    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:06.937584 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:06.937511    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:38:06.957236 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:06.957199    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:07.089579 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.089548    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twx5k\" (UniqueName: \"kubernetes.io/projected/bf908481-41c0-40b5-b5a9-7c3e885f83c7-kube-api-access-twx5k\") pod \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") "
Sep 16 21:38:07.089763 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.089604    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-metrics-certs\") pod \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") "
Sep 16 21:38:07.089763 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.089665    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf908481-41c0-40b5-b5a9-7c3e885f83c7-openshift-service-ca\") pod \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") "
Sep 16 21:38:07.089763 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.089752    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-maas-api-tls\") pod \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\" (UID: \"bf908481-41c0-40b5-b5a9-7c3e885f83c7\") "
Sep 16 21:38:07.090096 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.090065    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf908481-41c0-40b5-b5a9-7c3e885f83c7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bf908481-41c0-40b5-b5a9-7c3e885f83c7" (UID: "bf908481-41c0-40b5-b5a9-7c3e885f83c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:38:07.091722 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.091697    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bf908481-41c0-40b5-b5a9-7c3e885f83c7" (UID: "bf908481-41c0-40b5-b5a9-7c3e885f83c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:38:07.091834 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.091721    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bf908481-41c0-40b5-b5a9-7c3e885f83c7" (UID: "bf908481-41c0-40b5-b5a9-7c3e885f83c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:38:07.091834 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.091779    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf908481-41c0-40b5-b5a9-7c3e885f83c7-kube-api-access-twx5k" (OuterVolumeSpecName: "kube-api-access-twx5k") pod "bf908481-41c0-40b5-b5a9-7c3e885f83c7" (UID: "bf908481-41c0-40b5-b5a9-7c3e885f83c7"). InnerVolumeSpecName "kube-api-access-twx5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:38:07.191647 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.191553    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf908481-41c0-40b5-b5a9-7c3e885f83c7-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:38:07.191647 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.191587    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:38:07.191647 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.191596    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twx5k\" (UniqueName: \"kubernetes.io/projected/bf908481-41c0-40b5-b5a9-7c3e885f83c7-kube-api-access-twx5k\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:38:07.191647 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.191604    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf908481-41c0-40b5-b5a9-7c3e885f83c7-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:38:07.247941 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.247884    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:07.400829 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.400802    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:07.746747 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.746714    2567 generic.go:356] "Generic (PLEG): container finished" podID="bf908481-41c0-40b5-b5a9-7c3e885f83c7" containerID="dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc" exitCode=0
Sep 16 21:38:07.747164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.746761    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"
Sep 16 21:38:07.747164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.746797    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb" event={"ID":"bf908481-41c0-40b5-b5a9-7c3e885f83c7","Type":"ContainerDied","Data":"dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc"}
Sep 16 21:38:07.747164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.746873    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb" event={"ID":"bf908481-41c0-40b5-b5a9-7c3e885f83c7","Type":"ContainerDied","Data":"ebf6d3bafc754bb4b3a2a1f5777ae5380114e381230c1da58dcb9ecc3eb61c2f"}
Sep 16 21:38:07.747164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.746911    2567 scope.go:117] "RemoveContainer" containerID="dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc"
Sep 16 21:38:07.758968 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.758931    2567 scope.go:117] "RemoveContainer" containerID="dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc"
Sep 16 21:38:07.759343 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:38:07.759323    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc\": container with ID starting with dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc not found: ID does not exist" containerID="dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc"
Sep 16 21:38:07.759454 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.759351    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc"} err="failed to get container status \"dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc\": rpc error: code = NotFound desc = could not find container \"dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc\": container with ID starting with dc0f38bd9bcab3d9f988bac299944103354d0a8b11dfccc9a9f555d11d307cfc not found: ID does not exist"
Sep 16 21:38:07.768583 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.768558    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"]
Sep 16 21:38:07.772315 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.772293    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91829-548798795d-xvrzb"]
Sep 16 21:38:07.942999 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:07.942966    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf908481-41c0-40b5-b5a9-7c3e885f83c7" path="/var/lib/kubelet/pods/bf908481-41c0-40b5-b5a9-7c3e885f83c7/volumes"
Sep 16 21:38:08.130119 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:08.130081    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:09.053755 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:09.053719    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:09.962868 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:09.962834    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:10.666744 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:10.666703    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:12.045677 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:12.045646    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:12.149158 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:12.149124    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:12.251148 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:12.251120    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:12.347611 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:12.347534    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:13.086683 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:13.086651    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:13.183807 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:13.183769    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:14.230850 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:14.230818    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:14.331456 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:14.331427    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:14.504209 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:14.504164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:14.758346 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:14.758257    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:15.010150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:15.010056    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:15.260446 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:15.260352    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:15.443597 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:15.443556    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:16.536572 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:16.536489    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:16.635404 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:16.635373    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:16.736248 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:16.736212    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:20.560516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:20.560473    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:20.661404 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:20.661362    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:21.215144 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:21.215112    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:22.087793 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:22.087760    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:27.136950 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:27.136886    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:27.241316 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:27.241280    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:27.374088 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:27.374052    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:27.672527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:27.672481    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:27.771962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:27.771885    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:30.694405 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:30.694354    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:33.124094 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:33.124061    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:33.753030 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:33.752993    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:33.850629 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:33.850590    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:34.135822 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:34.135737    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:34.924787 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:34.924733    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:35.629490 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:35.629441    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:36.444381 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:36.444339    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:37.198673 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.198643    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"]
Sep 16 21:38:37.199349 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.199326    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf908481-41c0-40b5-b5a9-7c3e885f83c7" containerName="maas-api"
Sep 16 21:38:37.199563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.199547    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf908481-41c0-40b5-b5a9-7c3e885f83c7" containerName="maas-api"
Sep 16 21:38:37.199844 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.199825    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf908481-41c0-40b5-b5a9-7c3e885f83c7" containerName="maas-api"
Sep 16 21:38:37.203318 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.203296    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.205290 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.205268    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-06ac85db-serving-cert\""
Sep 16 21:38:37.216571 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.216544    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"]
Sep 16 21:38:37.231568 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.231529    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:37.282118 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.282082    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-maas-api-tls\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.282300 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.282173    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvn6l\" (UniqueName: \"kubernetes.io/projected/d08cf20a-bb4b-4811-816d-a48b774ecbff-kube-api-access-tvn6l\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.282300 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.282213    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-metrics-certs\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.282300 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.282251    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08cf20a-bb4b-4811-816d-a48b774ecbff-openshift-service-ca\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.289391 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.289358    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:37.383059 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.383018    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-maas-api-tls\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.383283 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.383082    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvn6l\" (UniqueName: \"kubernetes.io/projected/d08cf20a-bb4b-4811-816d-a48b774ecbff-kube-api-access-tvn6l\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.383283 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:38:37.383190    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-serving-cert: secret "maas-api-e2e-mt-06ac85db-serving-cert" not found
Sep 16 21:38:37.383283 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.383260    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-metrics-certs\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.383283 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:38:37.383271    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-maas-api-tls podName:d08cf20a-bb4b-4811-816d-a48b774ecbff nodeName:}" failed. No retries permitted until 2026-09-16 21:38:37.883250891 +0000 UTC m=+1004.531662870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-maas-api-tls") pod "maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw" (UID: "d08cf20a-bb4b-4811-816d-a48b774ecbff") : secret "maas-api-e2e-mt-06ac85db-serving-cert" not found
Sep 16 21:38:37.383520 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.383359    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08cf20a-bb4b-4811-816d-a48b774ecbff-openshift-service-ca\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.384126 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.384103    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08cf20a-bb4b-4811-816d-a48b774ecbff-openshift-service-ca\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.385970 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.385940    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-metrics-certs\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.391362 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.391333    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvn6l\" (UniqueName: \"kubernetes.io/projected/d08cf20a-bb4b-4811-816d-a48b774ecbff-kube-api-access-tvn6l\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.822080 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.822048    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:37.889068 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.889022    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-maas-api-tls\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.891589 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.891555    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-maas-api-tls\") pod \"maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:37.918418 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:37.918385    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:38:38.117398 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:38.117304    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:38.398212 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:38.398186    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"]
Sep 16 21:38:38.400634 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:38:38.400601    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08cf20a_bb4b_4811_816d_a48b774ecbff.slice/crio-e00831067466b38df9eb826bd17589c8eeae562451b0a0280b4ad47efd59783f WatchSource:0}: Error finding container e00831067466b38df9eb826bd17589c8eeae562451b0a0280b4ad47efd59783f: Status 404 returned error can't find the container with id e00831067466b38df9eb826bd17589c8eeae562451b0a0280b4ad47efd59783f
Sep 16 21:38:38.402054 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:38.402033    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:38:38.910465 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:38.910427    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw" event={"ID":"d08cf20a-bb4b-4811-816d-a48b774ecbff","Type":"ContainerStarted","Data":"88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c"}
Sep 16 21:38:38.910644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:38.910490    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:38.910644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:38.910507    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw" event={"ID":"d08cf20a-bb4b-4811-816d-a48b774ecbff","Type":"ContainerStarted","Data":"e00831067466b38df9eb826bd17589c8eeae562451b0a0280b4ad47efd59783f"}
Sep 16 21:38:38.928701 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:38.928637    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw" podStartSLOduration=1.552502349 podStartE2EDuration="1.928618988s" totalImagesPullingTime="376.116639ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:38:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:38:38.928092595 +0000 UTC m=+1005.576504585" watchObservedRunningTime="2026-09-16 21:38:38.928618988 +0000 UTC m=+1005.577030976"
Sep 16 21:38:44.921150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:44.921122    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:38:59.526962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:38:59.526891    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:00.221727 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:00.221688    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:00.916630 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:00.916572    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:01.722171 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:01.722126    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:02.346733 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:02.346700    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:03.002599 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:03.002559    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:03.923204 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:03.923164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:04.568612 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:04.568577    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:04.666878 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:04.666844    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:05.025501 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:05.025470    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:09.508714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:09.508683    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:11.003349 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:11.003315    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:11.106150 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:11.106087    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:12.217229 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:12.217189    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:12.910782 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:12.910749    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:13.663585 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:13.663547    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:14.430808 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:14.430766    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:15.206739 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:15.206702    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:15.931532 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:15.931499    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:16.545759 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:16.545725    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:16.646302 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:16.646265    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:21.817168 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:21.817128    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"]
Sep 16 21:39:21.817803 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:21.817765    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw" podUID="d08cf20a-bb4b-4811-816d-a48b774ecbff" containerName="maas-api" containerID="cri-o://88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c" gracePeriod=30
Sep 16 21:39:21.848389 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:21.848348    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:21.921347 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:21.921315    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:21.982887 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:21.982835    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:22.047099 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.047006    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:22.076067 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.075989    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:39:22.110692 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.110652    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:22.133838 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.133808    2567 generic.go:356] "Generic (PLEG): container finished" podID="d08cf20a-bb4b-4811-816d-a48b774ecbff" containerID="88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c" exitCode=0
Sep 16 21:39:22.134051 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.133859    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"
Sep 16 21:39:22.134051 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.133887    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw" event={"ID":"d08cf20a-bb4b-4811-816d-a48b774ecbff","Type":"ContainerDied","Data":"88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c"}
Sep 16 21:39:22.134051 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.133956    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw" event={"ID":"d08cf20a-bb4b-4811-816d-a48b774ecbff","Type":"ContainerDied","Data":"e00831067466b38df9eb826bd17589c8eeae562451b0a0280b4ad47efd59783f"}
Sep 16 21:39:22.134051 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.133982    2567 scope.go:117] "RemoveContainer" containerID="88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c"
Sep 16 21:39:22.143274 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.143255    2567 scope.go:117] "RemoveContainer" containerID="88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c"
Sep 16 21:39:22.143598 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:39:22.143562    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c\": container with ID starting with 88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c not found: ID does not exist" containerID="88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c"
Sep 16 21:39:22.143680 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.143610    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c"} err="failed to get container status \"88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c\": rpc error: code = NotFound desc = could not find container \"88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c\": container with ID starting with 88787e0a37b5382b9ac48cc55f8d48a125746bb162b1ad4e61c10e850c58036c not found: ID does not exist"
Sep 16 21:39:22.231054 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.231018    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-metrics-certs\") pod \"d08cf20a-bb4b-4811-816d-a48b774ecbff\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") "
Sep 16 21:39:22.231246 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.231105    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08cf20a-bb4b-4811-816d-a48b774ecbff-openshift-service-ca\") pod \"d08cf20a-bb4b-4811-816d-a48b774ecbff\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") "
Sep 16 21:39:22.231246 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.231136    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-maas-api-tls\") pod \"d08cf20a-bb4b-4811-816d-a48b774ecbff\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") "
Sep 16 21:39:22.231246 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.231159    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvn6l\" (UniqueName: \"kubernetes.io/projected/d08cf20a-bb4b-4811-816d-a48b774ecbff-kube-api-access-tvn6l\") pod \"d08cf20a-bb4b-4811-816d-a48b774ecbff\" (UID: \"d08cf20a-bb4b-4811-816d-a48b774ecbff\") "
Sep 16 21:39:22.231508 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.231476    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08cf20a-bb4b-4811-816d-a48b774ecbff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d08cf20a-bb4b-4811-816d-a48b774ecbff" (UID: "d08cf20a-bb4b-4811-816d-a48b774ecbff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:22.233368 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.233339    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08cf20a-bb4b-4811-816d-a48b774ecbff-kube-api-access-tvn6l" (OuterVolumeSpecName: "kube-api-access-tvn6l") pod "d08cf20a-bb4b-4811-816d-a48b774ecbff" (UID: "d08cf20a-bb4b-4811-816d-a48b774ecbff"). InnerVolumeSpecName "kube-api-access-tvn6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:22.233456 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.233334    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d08cf20a-bb4b-4811-816d-a48b774ecbff" (UID: "d08cf20a-bb4b-4811-816d-a48b774ecbff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:39:22.233456 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.233371    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d08cf20a-bb4b-4811-816d-a48b774ecbff" (UID: "d08cf20a-bb4b-4811-816d-a48b774ecbff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:39:22.332291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.332207    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:39:22.332291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.332236    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvn6l\" (UniqueName: \"kubernetes.io/projected/d08cf20a-bb4b-4811-816d-a48b774ecbff-kube-api-access-tvn6l\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:39:22.332291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.332246    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d08cf20a-bb4b-4811-816d-a48b774ecbff-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:39:22.332291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.332256    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d08cf20a-bb4b-4811-816d-a48b774ecbff-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:39:22.389231 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.389195    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:22.455537 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.455508    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"]
Sep 16 21:39:22.459663 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.459631    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:22.462188 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:22.462164    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06ac85db-5b6c66cd4f-mtfnw"]
Sep 16 21:39:23.229682 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:23.229652    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:23.944340 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:23.944306    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08cf20a-bb4b-4811-816d-a48b774ecbff" path="/var/lib/kubelet/pods/d08cf20a-bb4b-4811-816d-a48b774ecbff/volumes"
Sep 16 21:39:24.028972 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:24.028929    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:24.718690 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:24.718655    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:25.429174 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:25.429131    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:26.085561 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:26.085527    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:28.240786 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:28.240747    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:28.341706 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:28.341668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:28.501991 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:28.501955    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:28.873486 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:28.873398    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:28.977461 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:28.977418    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:29.105360 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:29.105327    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:34.687064 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:34.687025    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:34.752737 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:34.752688    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:34.860521 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:34.860488    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:34.950006 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:34.949881    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:35.070346 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:35.070310    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:37.373269 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.373236    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"]
Sep 16 21:39:37.373696 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.373683    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08cf20a-bb4b-4811-816d-a48b774ecbff" containerName="maas-api"
Sep 16 21:39:37.373696 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.373698    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08cf20a-bb4b-4811-816d-a48b774ecbff" containerName="maas-api"
Sep 16 21:39:37.373787 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.373771    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08cf20a-bb4b-4811-816d-a48b774ecbff" containerName="maas-api"
Sep 16 21:39:37.378250 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.378225    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.380981 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.380960    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-35f5fd\"/\"distinct-w0-35f5fd-kserve-self-signed-certs\""
Sep 16 21:39:37.382631 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.382609    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-home\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.382734 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.382646    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k847p\" (UniqueName: \"kubernetes.io/projected/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-kube-api-access-k847p\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.382734 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.382728    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tls-certs\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.382858 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.382839    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-model-cache\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.382939 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.382922    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-dshm\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.382993 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.382972    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tmp-dir\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.389218 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.389195    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"]
Sep 16 21:39:37.412443 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.412410    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:37.483743 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.483704    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tls-certs\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.483983 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.483774    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-model-cache\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.483983 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.483808    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-dshm\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.483983 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.483831    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tmp-dir\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.484176 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.484035    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-home\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.484176 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.484073    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k847p\" (UniqueName: \"kubernetes.io/projected/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-kube-api-access-k847p\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.484280 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.484239    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tmp-dir\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.484329 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.484295    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-model-cache\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.484404 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.484386    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-home\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.486209 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.486180    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-dshm\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.486604 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.486584    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tls-certs\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.492548 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.492523    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k847p\" (UniqueName: \"kubernetes.io/projected/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-kube-api-access-k847p\") pod \"distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") " pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.688862 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.688771    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:37.966122 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:37.966095    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"]
Sep 16 21:39:37.968065 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:39:37.968036    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc54f2f1d_ea4f_4917_bd0b_fd0bbf94f89a.slice/crio-2da74a107fb3d304de1c9df2254f6e637661089729ed20caf6924b5ef8cbe1f9 WatchSource:0}: Error finding container 2da74a107fb3d304de1c9df2254f6e637661089729ed20caf6924b5ef8cbe1f9: Status 404 returned error can't find the container with id 2da74a107fb3d304de1c9df2254f6e637661089729ed20caf6924b5ef8cbe1f9
Sep 16 21:39:38.220576 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:38.220485    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q" event={"ID":"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a","Type":"ContainerStarted","Data":"0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11"}
Sep 16 21:39:38.220576 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:38.220529    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q" event={"ID":"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a","Type":"ContainerStarted","Data":"2da74a107fb3d304de1c9df2254f6e637661089729ed20caf6924b5ef8cbe1f9"}
Sep 16 21:39:38.238276 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:38.238217    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q" podStartSLOduration=1.238199694 podStartE2EDuration="1.238199694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:39:38.23751228 +0000 UTC m=+1064.885924268" watchObservedRunningTime="2026-09-16 21:39:38.238199694 +0000 UTC m=+1064.886611684"
Sep 16 21:39:39.881049 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:39.880995    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:41.153326 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:41.153285    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:41.249826 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:41.249787    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:41.719279 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:41.719241    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:41.824205 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:41.824174    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:42.093878 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:42.093840    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:42.226637 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:42.226600    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:42.325638 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:42.325599    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:42.653216 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:42.653174    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:43.080155 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:43.080121    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:43.883345 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:43.883308    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:44.714134 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:44.714072    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:45.522741 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:45.522703    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:45.800472 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:45.800381    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:45.896986 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:45.896939    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:46.430643 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:46.430594    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:47.309996 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:47.309958    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:47.410724 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:47.410684    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:47.689815 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:47.689723    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:47.690033 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:47.689869    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:47.702522 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:47.702495    2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:48.106174 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:48.106141    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:48.291181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:48.291154    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:39:49.116028 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.115995    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"]
Sep 16 21:39:49.120053 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.120028    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.122779 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.122754    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-296002-serving-cert\""
Sep 16 21:39:49.133007 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.132967    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"]
Sep 16 21:39:49.166988 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.166949    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:49.203011 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.202976    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.203186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.203028    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnflx\" (UniqueName: \"kubernetes.io/projected/babb773c-f5e2-4c96-9b02-53db1eda5f6a-kube-api-access-mnflx\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.203186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.203107    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.203266 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.203178    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/babb773c-f5e2-4c96-9b02-53db1eda5f6a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.272799 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.272759    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:49.304188 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.304154    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/babb773c-f5e2-4c96-9b02-53db1eda5f6a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.304363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.304223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.304363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.304270    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnflx\" (UniqueName: \"kubernetes.io/projected/babb773c-f5e2-4c96-9b02-53db1eda5f6a-kube-api-access-mnflx\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.304363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.304349    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.304541 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:39:49.304356    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-serving-cert: secret "maas-api-e2e-shared-a-w2-296002-serving-cert" not found
Sep 16 21:39:49.304541 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:39:49.304430    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-maas-api-tls podName:babb773c-f5e2-4c96-9b02-53db1eda5f6a nodeName:}" failed. No retries permitted until 2026-09-16 21:39:49.804409555 +0000 UTC m=+1076.452821522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-maas-api-tls") pod "maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb" (UID: "babb773c-f5e2-4c96-9b02-53db1eda5f6a") : secret "maas-api-e2e-shared-a-w2-296002-serving-cert" not found
Sep 16 21:39:49.305002 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.304981    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/babb773c-f5e2-4c96-9b02-53db1eda5f6a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.306837 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.306816    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.313523 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.313487    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnflx\" (UniqueName: \"kubernetes.io/projected/babb773c-f5e2-4c96-9b02-53db1eda5f6a-kube-api-access-mnflx\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.644500 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.644465    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:49.748721 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.748681    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:49.810053 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.810015    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.812697 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.812668    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:49.828349 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:49.828302    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:50.033607 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:50.033569    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:50.036055 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:50.036018    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:50.307683 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:50.307655    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"]
Sep 16 21:39:50.309576 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:39:50.309548    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabb773c_f5e2_4c96_9b02_53db1eda5f6a.slice/crio-ae29390e7a649aa9b8a360326a2ad61a0b2e00d785f7182345551d98130e858d WatchSource:0}: Error finding container ae29390e7a649aa9b8a360326a2ad61a0b2e00d785f7182345551d98130e858d: Status 404 returned error can't find the container with id ae29390e7a649aa9b8a360326a2ad61a0b2e00d785f7182345551d98130e858d
Sep 16 21:39:50.737215 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:50.737180    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:51.297807 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:51.297768    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb" event={"ID":"babb773c-f5e2-4c96-9b02-53db1eda5f6a","Type":"ContainerStarted","Data":"84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d"}
Sep 16 21:39:51.298040 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:51.297827    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb" event={"ID":"babb773c-f5e2-4c96-9b02-53db1eda5f6a","Type":"ContainerStarted","Data":"ae29390e7a649aa9b8a360326a2ad61a0b2e00d785f7182345551d98130e858d"}
Sep 16 21:39:51.298040 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:51.297989    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:51.319548 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:51.319493    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb" podStartSLOduration=1.961092368 podStartE2EDuration="2.319475482s" totalImagesPullingTime="358.383114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:39:51.317528683 +0000 UTC m=+1077.965940672" watchObservedRunningTime="2026-09-16 21:39:51.319475482 +0000 UTC m=+1077.967887515"
Sep 16 21:39:51.550698 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:51.550600    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:52.633592 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:52.633554    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:52.691043 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:52.691005    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:54.137486 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:54.137448    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:54.671157 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:54.671120    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"]
Sep 16 21:39:54.671460 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:54.671409    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv" podUID="eb04fa2b-2476-4513-bafe-5ab497525b31" containerName="maas-api" containerID="cri-o://cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58" gracePeriod=30
Sep 16 21:39:54.703089 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:54.703051    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:54.803363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:54.803334    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:54.903765 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:54.903733    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:54.928772 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:54.928710    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:39:54.973091 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:54.973051    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:55.063005 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.062960    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcmcl\" (UniqueName: \"kubernetes.io/projected/eb04fa2b-2476-4513-bafe-5ab497525b31-kube-api-access-vcmcl\") pod \"eb04fa2b-2476-4513-bafe-5ab497525b31\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") "
Sep 16 21:39:55.063194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.063100    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-maas-api-tls\") pod \"eb04fa2b-2476-4513-bafe-5ab497525b31\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") "
Sep 16 21:39:55.063194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.063140    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb04fa2b-2476-4513-bafe-5ab497525b31-openshift-service-ca\") pod \"eb04fa2b-2476-4513-bafe-5ab497525b31\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") "
Sep 16 21:39:55.063194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.063175    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-metrics-certs\") pod \"eb04fa2b-2476-4513-bafe-5ab497525b31\" (UID: \"eb04fa2b-2476-4513-bafe-5ab497525b31\") "
Sep 16 21:39:55.063839 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.063806    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb04fa2b-2476-4513-bafe-5ab497525b31-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eb04fa2b-2476-4513-bafe-5ab497525b31" (UID: "eb04fa2b-2476-4513-bafe-5ab497525b31"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:39:55.065730 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.065704    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb04fa2b-2476-4513-bafe-5ab497525b31-kube-api-access-vcmcl" (OuterVolumeSpecName: "kube-api-access-vcmcl") pod "eb04fa2b-2476-4513-bafe-5ab497525b31" (UID: "eb04fa2b-2476-4513-bafe-5ab497525b31"). InnerVolumeSpecName "kube-api-access-vcmcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:39:55.066170 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.066132    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eb04fa2b-2476-4513-bafe-5ab497525b31" (UID: "eb04fa2b-2476-4513-bafe-5ab497525b31"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:39:55.066287 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.066253    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eb04fa2b-2476-4513-bafe-5ab497525b31" (UID: "eb04fa2b-2476-4513-bafe-5ab497525b31"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:39:55.165169 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.165131    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmcl\" (UniqueName: \"kubernetes.io/projected/eb04fa2b-2476-4513-bafe-5ab497525b31-kube-api-access-vcmcl\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.165169 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.165169    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.165575 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.165185    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb04fa2b-2476-4513-bafe-5ab497525b31-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.165575 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.165199    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb04fa2b-2476-4513-bafe-5ab497525b31-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:39:55.319791 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.319755    2567 generic.go:356] "Generic (PLEG): container finished" podID="eb04fa2b-2476-4513-bafe-5ab497525b31" containerID="cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58" exitCode=0
Sep 16 21:39:55.319962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.319809    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"
Sep 16 21:39:55.319962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.319821    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv" event={"ID":"eb04fa2b-2476-4513-bafe-5ab497525b31","Type":"ContainerDied","Data":"cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58"}
Sep 16 21:39:55.319962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.319870    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv" event={"ID":"eb04fa2b-2476-4513-bafe-5ab497525b31","Type":"ContainerDied","Data":"719b375f4f3db405b6af7b29eefc3870f6691290d22d6bb7ed58ea1a03201631"}
Sep 16 21:39:55.319962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.319891    2567 scope.go:117] "RemoveContainer" containerID="cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58"
Sep 16 21:39:55.329626 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.329608    2567 scope.go:117] "RemoveContainer" containerID="cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58"
Sep 16 21:39:55.329893 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:39:55.329874    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58\": container with ID starting with cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58 not found: ID does not exist" containerID="cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58"
Sep 16 21:39:55.329986 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.329921    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58"} err="failed to get container status \"cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58\": rpc error: code = NotFound desc = could not find container \"cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58\": container with ID starting with cdf9f4c0646c1acc71d22a8ac7b81d2b976548e85df8063e0bfc80c05dca2d58 not found: ID does not exist"
Sep 16 21:39:55.343828 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.343801    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"]
Sep 16 21:39:55.346250 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.346226    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e5568d-6449db59f-7tjbv"]
Sep 16 21:39:55.385479 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.385399    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:55.486350 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.486311    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:55.943199 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:55.943163    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb04fa2b-2476-4513-bafe-5ab497525b31" path="/var/lib/kubelet/pods/eb04fa2b-2476-4513-bafe-5ab497525b31/volumes"
Sep 16 21:39:56.055178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:56.055143    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:56.727140 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:56.727106    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:57.308178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:57.308146    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:39:57.591379 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:57.591285    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:39:58.437400 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:39:58.437360    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:01.804969 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:01.804876    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:01.890411 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:01.890373    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:02.097740 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:02.097636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:02.542447 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:02.542412    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:03.310824 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:03.310773    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:04.024723 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:04.024679    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:04.718499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:04.718463    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:05.583732 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:05.583676    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:05.688283 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:05.688253    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:06.488311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:06.488278    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:07.137054 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:07.137011    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:07.927892 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:07.927860    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:08.653189 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:08.653158    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:09.640670 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:09.640632    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:10.742162 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:10.742126    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:10.866522 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:10.866479    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:11.217915 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:11.217864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:11.466282 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:11.466236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:11.568670 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:11.568633    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:11.802667 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:11.802635    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:12.685310 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:12.685278    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:13.240754 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.240717    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"]
Sep 16 21:40:13.241264 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.240975    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x" podUID="081c381f-9818-4f06-831b-64ed0fa73fe8" containerName="maas-api" containerID="cri-o://b6869a9eab9572e576155c87e66d7170a3fb0552534c71c461eefaa540191e75" gracePeriod=30
Sep 16 21:40:13.267371 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.267340    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:13.368832 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.368798    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:13.427954 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.427894    2567 generic.go:356] "Generic (PLEG): container finished" podID="081c381f-9818-4f06-831b-64ed0fa73fe8" containerID="b6869a9eab9572e576155c87e66d7170a3fb0552534c71c461eefaa540191e75" exitCode=0
Sep 16 21:40:13.428153 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.427955    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x" event={"ID":"081c381f-9818-4f06-831b-64ed0fa73fe8","Type":"ContainerDied","Data":"b6869a9eab9572e576155c87e66d7170a3fb0552534c71c461eefaa540191e75"}
Sep 16 21:40:13.457742 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.457683    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:13.495284 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.495220    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:40:13.518037 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.518008    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:13.549790 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.549759    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081c381f-9818-4f06-831b-64ed0fa73fe8-openshift-service-ca\") pod \"081c381f-9818-4f06-831b-64ed0fa73fe8\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") "
Sep 16 21:40:13.549993 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.549818    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t2v8\" (UniqueName: \"kubernetes.io/projected/081c381f-9818-4f06-831b-64ed0fa73fe8-kube-api-access-8t2v8\") pod \"081c381f-9818-4f06-831b-64ed0fa73fe8\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") "
Sep 16 21:40:13.549993 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.549863    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-maas-api-tls\") pod \"081c381f-9818-4f06-831b-64ed0fa73fe8\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") "
Sep 16 21:40:13.549993 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.549879    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-metrics-certs\") pod \"081c381f-9818-4f06-831b-64ed0fa73fe8\" (UID: \"081c381f-9818-4f06-831b-64ed0fa73fe8\") "
Sep 16 21:40:13.550212 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.550180    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081c381f-9818-4f06-831b-64ed0fa73fe8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "081c381f-9818-4f06-831b-64ed0fa73fe8" (UID: "081c381f-9818-4f06-831b-64ed0fa73fe8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:40:13.550293 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.550279    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/081c381f-9818-4f06-831b-64ed0fa73fe8-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:40:13.552186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.552151    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "081c381f-9818-4f06-831b-64ed0fa73fe8" (UID: "081c381f-9818-4f06-831b-64ed0fa73fe8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:40:13.552186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.552176    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "081c381f-9818-4f06-831b-64ed0fa73fe8" (UID: "081c381f-9818-4f06-831b-64ed0fa73fe8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:40:13.552375 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.552229    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081c381f-9818-4f06-831b-64ed0fa73fe8-kube-api-access-8t2v8" (OuterVolumeSpecName: "kube-api-access-8t2v8") pod "081c381f-9818-4f06-831b-64ed0fa73fe8" (UID: "081c381f-9818-4f06-831b-64ed0fa73fe8"). InnerVolumeSpecName "kube-api-access-8t2v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:40:13.651001 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.650958    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t2v8\" (UniqueName: \"kubernetes.io/projected/081c381f-9818-4f06-831b-64ed0fa73fe8-kube-api-access-8t2v8\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:40:13.651001 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.650997    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:40:13.651293 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.651013    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/081c381f-9818-4f06-831b-64ed0fa73fe8-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:40:13.842788 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.842735    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:13.945686 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:13.945619    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:14.434613 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:14.434581    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"
Sep 16 21:40:14.435099 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:14.434618    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x" event={"ID":"081c381f-9818-4f06-831b-64ed0fa73fe8","Type":"ContainerDied","Data":"286686eaceba51866643669a116408d0da21c2d57f73c56afbaa439e5e6bae6e"}
Sep 16 21:40:14.435099 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:14.434691    2567 scope.go:117] "RemoveContainer" containerID="b6869a9eab9572e576155c87e66d7170a3fb0552534c71c461eefaa540191e75"
Sep 16 21:40:14.450851 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:14.450826    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"]
Sep 16 21:40:14.454413 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:14.454393    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-44e517-d69bb6c4c-fnj5x"]
Sep 16 21:40:14.750593 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:14.750557    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:15.056533 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:15.056429    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:15.286022 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:15.285976    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:15.943260 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:15.943223    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="081c381f-9818-4f06-831b-64ed0fa73fe8" path="/var/lib/kubelet/pods/081c381f-9818-4f06-831b-64ed0fa73fe8/volumes"
Sep 16 21:40:16.129700 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:16.129665    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:16.591637 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:16.591561    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:16.960024 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:16.959931    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:18.004809 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.004765    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"]
Sep 16 21:40:18.005451 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.005434    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb04fa2b-2476-4513-bafe-5ab497525b31" containerName="maas-api"
Sep 16 21:40:18.005506 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.005455    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb04fa2b-2476-4513-bafe-5ab497525b31" containerName="maas-api"
Sep 16 21:40:18.005506 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.005485    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081c381f-9818-4f06-831b-64ed0fa73fe8" containerName="maas-api"
Sep 16 21:40:18.005506 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.005494    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="081c381f-9818-4f06-831b-64ed0fa73fe8" containerName="maas-api"
Sep 16 21:40:18.005617 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.005605    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="081c381f-9818-4f06-831b-64ed0fa73fe8" containerName="maas-api"
Sep 16 21:40:18.005657 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.005619    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb04fa2b-2476-4513-bafe-5ab497525b31" containerName="maas-api"
Sep 16 21:40:18.008520 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.008503    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.011199 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.011176    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-365ea1-serving-cert\""
Sep 16 21:40:18.020051 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.020024    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"]
Sep 16 21:40:18.040555 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.040515    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:18.094582 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.094549    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.094582 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.094587    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfqrq\" (UniqueName: \"kubernetes.io/projected/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-kube-api-access-mfqrq\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.094798 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.094629    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.094798 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.094714    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.101882 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.101821    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:18.195414 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.195375    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.195414 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.195416    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfqrq\" (UniqueName: \"kubernetes.io/projected/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-kube-api-access-mfqrq\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.195634 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.195459    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.195634 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.195497    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.195634 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:40:18.195622    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-serving-cert: secret "maas-api-e2e-shared-b-w2-365ea1-serving-cert" not found
Sep 16 21:40:18.195757 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:40:18.195701    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls podName:0a4f5ad9-50ac-4d63-80e3-3e474a1040b2 nodeName:}" failed. No retries permitted until 2026-09-16 21:40:18.695679736 +0000 UTC m=+1105.344091715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls") pod "maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp" (UID: "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2") : secret "maas-api-e2e-shared-b-w2-365ea1-serving-cert" not found
Sep 16 21:40:18.196253 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.196229    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.198054 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.198034    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.204429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.204411    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfqrq\" (UniqueName: \"kubernetes.io/projected/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-kube-api-access-mfqrq\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.510309 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.510274    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:18.613592 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.613553    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:18.701241 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.701206    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:18.701429 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:40:18.701350    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-serving-cert: secret "maas-api-e2e-shared-b-w2-365ea1-serving-cert" not found
Sep 16 21:40:18.701493 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:40:18.701428    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls podName:0a4f5ad9-50ac-4d63-80e3-3e474a1040b2 nodeName:}" failed. No retries permitted until 2026-09-16 21:40:19.701413918 +0000 UTC m=+1106.349825884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls") pod "maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp" (UID: "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2") : secret "maas-api-e2e-shared-b-w2-365ea1-serving-cert" not found
Sep 16 21:40:18.782017 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.781890    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:18.972984 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:18.972946    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:19.710453 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:19.710419    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:19.712875 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:19.712853    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:19.822562 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:19.822534    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:19.825075 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:19.824813    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:20.103999 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:20.103972    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"]
Sep 16 21:40:20.105991 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:40:20.105950    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4f5ad9_50ac_4d63_80e3_3e474a1040b2.slice/crio-c94da0ca09567cb41ef53d74118b7801ca9a0aeca291fd7ad3a0b25ff60194b0 WatchSource:0}: Error finding container c94da0ca09567cb41ef53d74118b7801ca9a0aeca291fd7ad3a0b25ff60194b0: Status 404 returned error can't find the container with id c94da0ca09567cb41ef53d74118b7801ca9a0aeca291fd7ad3a0b25ff60194b0
Sep 16 21:40:20.478124 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:20.478094    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp" event={"ID":"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2","Type":"ContainerStarted","Data":"c94da0ca09567cb41ef53d74118b7801ca9a0aeca291fd7ad3a0b25ff60194b0"}
Sep 16 21:40:20.875442 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:20.875361    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:20.977557 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:20.977514    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:21.163484 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:21.163391    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:21.485405 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:21.485311    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp" event={"ID":"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2","Type":"ContainerStarted","Data":"55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f"}
Sep 16 21:40:21.485405 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:21.485386    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:21.503861 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:21.503804    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp" podStartSLOduration=4.190719273 podStartE2EDuration="4.503785146s" totalImagesPullingTime="313.065873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:21.501839806 +0000 UTC m=+1108.150251794" watchObservedRunningTime="2026-09-16 21:40:21.503785146 +0000 UTC m=+1108.152197135"
Sep 16 21:40:21.694398 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:21.694361    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:22.019572 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:22.019529    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:22.156634 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:22.156600    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:24.113912 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:24.113867    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:24.289258 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:24.289224    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:24.906526 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:24.906488    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:25.018300 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:25.018267    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:25.592371 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:25.592332    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:26.537720 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:26.537686    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:27.347309 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:27.347272    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:27.495514 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:27.495488    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:40:28.250103 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:28.250071    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:29.011651 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.011611    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"]
Sep 16 21:40:29.015551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.015527    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.024971 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.024840    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a7df81-serving-cert\""
Sep 16 21:40:29.035999 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.035969    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"]
Sep 16 21:40:29.061001 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.060958    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:29.111352 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.111319    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.111352 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.111361    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c159f43-095a-49db-af51-bb61d57258a6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.111613 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.111380    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zstct\" (UniqueName: \"kubernetes.io/projected/5c159f43-095a-49db-af51-bb61d57258a6-kube-api-access-zstct\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.111613 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.111470    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.150231 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.150197    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:29.212542 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.212509    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.212542 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.212543    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c159f43-095a-49db-af51-bb61d57258a6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.212760 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.212562    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zstct\" (UniqueName: \"kubernetes.io/projected/5c159f43-095a-49db-af51-bb61d57258a6-kube-api-access-zstct\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.212760 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.212600    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.212760 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:40:29.212658    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-serving-cert: secret "maas-api-e2e-shared-a-w3-a7df81-serving-cert" not found
Sep 16 21:40:29.212760 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:40:29.212732    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-maas-api-tls podName:5c159f43-095a-49db-af51-bb61d57258a6 nodeName:}" failed. No retries permitted until 2026-09-16 21:40:29.712712519 +0000 UTC m=+1116.361124499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct" (UID: "5c159f43-095a-49db-af51-bb61d57258a6") : secret "maas-api-e2e-shared-a-w3-a7df81-serving-cert" not found
Sep 16 21:40:29.213227 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.213208    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c159f43-095a-49db-af51-bb61d57258a6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.214950 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.214922    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.228012 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.227989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zstct\" (UniqueName: \"kubernetes.io/projected/5c159f43-095a-49db-af51-bb61d57258a6-kube-api-access-zstct\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.316457 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.316360    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:29.718175 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.718092    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.720342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.720322    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:29.747162 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.747114    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:29.841361 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.841324    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:29.927863 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:29.927829    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:30.204306 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:30.204280    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"]
Sep 16 21:40:30.206654 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:40:30.206616    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c159f43_095a_49db_af51_bb61d57258a6.slice/crio-762cfbb9b63a67cdda69561b2116f8825f7f8280b2b0fc212caadc34db1b96f8 WatchSource:0}: Error finding container 762cfbb9b63a67cdda69561b2116f8825f7f8280b2b0fc212caadc34db1b96f8: Status 404 returned error can't find the container with id 762cfbb9b63a67cdda69561b2116f8825f7f8280b2b0fc212caadc34db1b96f8
Sep 16 21:40:30.226278 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:30.226174    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:30.537472 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:30.537439    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct" event={"ID":"5c159f43-095a-49db-af51-bb61d57258a6","Type":"ContainerStarted","Data":"762cfbb9b63a67cdda69561b2116f8825f7f8280b2b0fc212caadc34db1b96f8"}
Sep 16 21:40:31.033163 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:31.033118    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:31.544720 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:31.544684    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct" event={"ID":"5c159f43-095a-49db-af51-bb61d57258a6","Type":"ContainerStarted","Data":"4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2"}
Sep 16 21:40:31.545146 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:31.544785    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:31.563183 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:31.563130    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct" podStartSLOduration=3.2267641400000002 podStartE2EDuration="3.563115146s" totalImagesPullingTime="336.351006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:31.561632681 +0000 UTC m=+1118.210044693" watchObservedRunningTime="2026-09-16 21:40:31.563115146 +0000 UTC m=+1118.211527192"
Sep 16 21:40:31.810500 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:31.810385    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:32.503831 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:32.503796    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:33.139123 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:33.139090    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:34.014831 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:34.014776    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:34.777138 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:34.777105    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:34.877307 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:34.877275    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:35.595447 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:35.595409    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:36.272412 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:36.272363    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:37.556624 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:37.556597    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:40:38.998783 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:38.998736    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:45.364918 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:45.364873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:45.646931 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:45.646836    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:45.814093 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:45.814054    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:48.363435 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:48.363396    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:49.179312 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:49.179275    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:49.327425 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:49.327384    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:49.485010 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:49.484926    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:50.821190 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:50.821155    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:51.554424 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:51.554388    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:52.446112 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:52.446078    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:53.213990 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:53.213948    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:54.096091 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.096056    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"]
Sep 16 21:40:54.101255 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.101227    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.103505 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.103480    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5ee44a89-serving-cert\""
Sep 16 21:40:54.109063 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.109039    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"]
Sep 16 21:40:54.123564 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.123515    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:54.155916 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.155871    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-maas-api-tls\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.156086 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.155981    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38487eee-a666-4184-bf5a-a6555765f16b-openshift-service-ca\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.156086 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.156026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpr9\" (UniqueName: \"kubernetes.io/projected/38487eee-a666-4184-bf5a-a6555765f16b-kube-api-access-tmpr9\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.156086 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.156062    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-metrics-certs\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.189955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.189838    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:54.256632 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.256601    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38487eee-a666-4184-bf5a-a6555765f16b-openshift-service-ca\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.256838 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.256657    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpr9\" (UniqueName: \"kubernetes.io/projected/38487eee-a666-4184-bf5a-a6555765f16b-kube-api-access-tmpr9\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.256838 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.256713    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-metrics-certs\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.256838 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.256779    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-maas-api-tls\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.257457 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.257427    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38487eee-a666-4184-bf5a-a6555765f16b-openshift-service-ca\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.259743 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.259722    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-metrics-certs\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.259870 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.259780    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-maas-api-tls\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.264724 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.264696    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpr9\" (UniqueName: \"kubernetes.io/projected/38487eee-a666-4184-bf5a-a6555765f16b-kube-api-access-tmpr9\") pod \"maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.290753 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.290726    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:54.414166 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.414135    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:54.644701 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.644663    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:54.686766 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.686741    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"]
Sep 16 21:40:54.688507 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:40:54.688481    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38487eee_a666_4184_bf5a_a6555765f16b.slice/crio-78c62a810b4d87d5d362449c514cfbcab8be52c84b1481f97a8fcdb6818f1f8c WatchSource:0}: Error finding container 78c62a810b4d87d5d362449c514cfbcab8be52c84b1481f97a8fcdb6818f1f8c: Status 404 returned error can't find the container with id 78c62a810b4d87d5d362449c514cfbcab8be52c84b1481f97a8fcdb6818f1f8c
Sep 16 21:40:54.750690 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:54.750662    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:55.049572 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:55.049479    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:55.150705 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:55.150664    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:55.394279 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:55.394187    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:55.679751 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:55.679663    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk" event={"ID":"38487eee-a666-4184-bf5a-a6555765f16b","Type":"ContainerStarted","Data":"4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa"}
Sep 16 21:40:55.680219 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:55.680191    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk" event={"ID":"38487eee-a666-4184-bf5a-a6555765f16b","Type":"ContainerStarted","Data":"78c62a810b4d87d5d362449c514cfbcab8be52c84b1481f97a8fcdb6818f1f8c"}
Sep 16 21:40:55.680401 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:55.680386    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:40:55.692635 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:55.692606    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:55.704360 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:55.704300    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk" podStartSLOduration=0.89721434 podStartE2EDuration="1.704277366s" totalImagesPullingTime="807.063026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:40:55.699874866 +0000 UTC m=+1142.348286856" watchObservedRunningTime="2026-09-16 21:40:55.704277366 +0000 UTC m=+1142.352689355"
Sep 16 21:40:55.936460 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:55.936366    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:56.137214 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:56.137181    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:56.629205 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:56.629153    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:57.324829 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:57.324796    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:58.140434 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:58.140400    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:58.949461 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:58.949421    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:59.295582 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:59.295547    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:40:59.394423 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:40:59.394386    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:00.061255 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.061221    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:00.805505 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.805472    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"]
Sep 16 21:41:00.810479 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.810452    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:00.813769 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.813748    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-1b68db-serving-cert\""
Sep 16 21:41:00.823295 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.823265    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"]
Sep 16 21:41:00.851219 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.851181    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:00.924654 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.924620    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:00.924654 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.924665    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ns2d\" (UniqueName: \"kubernetes.io/projected/320c01da-fb7b-4888-8ab2-3f7df946165e-kube-api-access-4ns2d\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:00.924935 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.924703    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320c01da-fb7b-4888-8ab2-3f7df946165e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:00.924935 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.924736    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:00.951566 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:00.951526    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:01.025992 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.025955    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:01.026188 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.026008    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ns2d\" (UniqueName: \"kubernetes.io/projected/320c01da-fb7b-4888-8ab2-3f7df946165e-kube-api-access-4ns2d\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:01.026188 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.026049    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320c01da-fb7b-4888-8ab2-3f7df946165e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:01.026188 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.026092    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:01.026749 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.026727    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320c01da-fb7b-4888-8ab2-3f7df946165e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:01.028487 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.028456    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:01.028587 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.028535    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:01.036397 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.036373    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ns2d\" (UniqueName: \"kubernetes.io/projected/320c01da-fb7b-4888-8ab2-3f7df946165e-kube-api-access-4ns2d\") pod \"maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:01.054356 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.054326    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:01.123463 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.123373    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:01.409720 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.409694    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"]
Sep 16 21:41:01.412260 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:41:01.412228    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320c01da_fb7b_4888_8ab2_3f7df946165e.slice/crio-c7b258ffb75af85346bd26525c02fbe8afe5c9fa748cdee175703718ad2dfb3d WatchSource:0}: Error finding container c7b258ffb75af85346bd26525c02fbe8afe5c9fa748cdee175703718ad2dfb3d: Status 404 returned error can't find the container with id c7b258ffb75af85346bd26525c02fbe8afe5c9fa748cdee175703718ad2dfb3d
Sep 16 21:41:01.442205 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.442172    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:01.540561 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.540517    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:01.586527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.586489    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"]
Sep 16 21:41:01.591178 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.591158    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:01.593702 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.593678    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-1b68db\""
Sep 16 21:41:01.593798 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.593744    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-1b68db-dockercfg-s68cp\""
Sep 16 21:41:01.603148 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.603126    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"]
Sep 16 21:41:01.630921 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.630866    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw2j6\" (UniqueName: \"kubernetes.io/projected/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-kube-api-access-rw2j6\") pod \"payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg\" (UID: \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:01.631048 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.630927    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg\" (UID: \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:01.713811 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.713722    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7" event={"ID":"320c01da-fb7b-4888-8ab2-3f7df946165e","Type":"ContainerStarted","Data":"c7b258ffb75af85346bd26525c02fbe8afe5c9fa748cdee175703718ad2dfb3d"}
Sep 16 21:41:01.732353 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.732316    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg\" (UID: \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:01.732520 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.732489    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw2j6\" (UniqueName: \"kubernetes.io/projected/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-kube-api-access-rw2j6\") pod \"payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg\" (UID: \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:01.733161 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.733145    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg\" (UID: \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:01.740420 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.740398    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw2j6\" (UniqueName: \"kubernetes.io/projected/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-kube-api-access-rw2j6\") pod \"payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg\" (UID: \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:01.851419 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.851392    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:01.904135 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:01.904096    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:02.179827 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.179734    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"]
Sep 16 21:41:02.182609 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:41:02.182578    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26d8bf9_fb0e_4bfe_8bd6_911e3ed4e4d7.slice/crio-6037e804fb8b7f7e7e002bbf324c8a2817797b37bf625a5bead2d1434ff41fd9 WatchSource:0}: Error finding container 6037e804fb8b7f7e7e002bbf324c8a2817797b37bf625a5bead2d1434ff41fd9: Status 404 returned error can't find the container with id 6037e804fb8b7f7e7e002bbf324c8a2817797b37bf625a5bead2d1434ff41fd9
Sep 16 21:41:02.696957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.696922    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:41:02.722216 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.722181    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7" event={"ID":"320c01da-fb7b-4888-8ab2-3f7df946165e","Type":"ContainerStarted","Data":"8cc9fc16604922e75e1959e7939d3cdc127473d62cfec00250f9f01783059c9a"}
Sep 16 21:41:02.722881 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.722852    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:02.724411 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.724381    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg" event={"ID":"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7","Type":"ContainerStarted","Data":"06eae18d20e63a661aa7855d84a0d1dcef30315a8b6fd8b276c1a60db0a4f6f9"}
Sep 16 21:41:02.724527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.724434    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg" event={"ID":"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7","Type":"ContainerStarted","Data":"6037e804fb8b7f7e7e002bbf324c8a2817797b37bf625a5bead2d1434ff41fd9"}
Sep 16 21:41:02.724621 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.724557    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:02.734615 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.734589    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:02.740866 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.740812    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7" podStartSLOduration=2.350312096 podStartE2EDuration="2.740793936s" totalImagesPullingTime="390.48184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:41:02.738758466 +0000 UTC m=+1149.387170467" watchObservedRunningTime="2026-09-16 21:41:02.740793936 +0000 UTC m=+1149.389205924"
Sep 16 21:41:02.758401 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:02.758347    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg" podStartSLOduration=1.758332263 podStartE2EDuration="1.758332263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:41:02.756391568 +0000 UTC m=+1149.404803551" watchObservedRunningTime="2026-09-16 21:41:02.758332263 +0000 UTC m=+1149.406744251"
Sep 16 21:41:03.421370 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:03.421331    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:04.224267 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:04.224228    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:04.936935 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:04.936863    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:05.632513 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:05.632480    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:06.305627 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:06.305577    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:08.739595 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:08.739567    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:41:13.735890 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:13.735861    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:41:15.018526 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:15.018464    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:16.220745 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:16.220647    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:16.875320 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:16.875261    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:17.708469 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:17.708430    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:18.582853 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:18.582810    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:19.198765 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:19.198725    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:19.564894 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:19.564857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:20.889769 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:20.889731    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:22.050259 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:22.050215    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:22.774343 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:22.774304    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:23.433995 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:23.433895    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:24.226461 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:24.226415    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:24.332581 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:24.332551    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:25.109748 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:25.109708    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:25.902627 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:25.902592    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:26.650614 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:26.650582    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:40.553318 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:40.553280    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:45.917443 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:45.917404    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:41:53.878887 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:53.878850    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:41:53.881647 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:53.881623    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:41:57.275266 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:41:57.275226    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:42:05.222611 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:42:05.222568    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:42:07.976677 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:42:07.976642    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:42:26.816878 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:42:26.816844    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:42:36.790598 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:42:36.790558    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:02.717840 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:02.717799    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:06.433778 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:06.433740    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:07.123652 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:07.123616    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:07.935734 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:07.935684    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:08.586428 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:08.586396    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:08.924511 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:08.924411    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:09.683704 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:09.683667    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:10.435936 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:10.435884    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:11.337701 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:11.337670    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:12.069276 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:12.069236    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:12.748209 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:12.748153    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:21.643417 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:21.643379    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"]
Sep 16 21:43:21.643873 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:21.643642    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk" podUID="38487eee-a666-4184-bf5a-a6555765f16b" containerName="maas-api" containerID="cri-o://4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa" gracePeriod=30
Sep 16 21:43:21.687887 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:21.687859    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:21.788271 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:21.788229    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:21.886780 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:21.886705    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:21.917105 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:21.917077    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:43:21.985454 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:21.985416    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:22.050660 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.050626    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmpr9\" (UniqueName: \"kubernetes.io/projected/38487eee-a666-4184-bf5a-a6555765f16b-kube-api-access-tmpr9\") pod \"38487eee-a666-4184-bf5a-a6555765f16b\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") "
Sep 16 21:43:22.050855 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.050682    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-maas-api-tls\") pod \"38487eee-a666-4184-bf5a-a6555765f16b\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") "
Sep 16 21:43:22.050855 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.050753    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38487eee-a666-4184-bf5a-a6555765f16b-openshift-service-ca\") pod \"38487eee-a666-4184-bf5a-a6555765f16b\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") "
Sep 16 21:43:22.050855 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.050824    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-metrics-certs\") pod \"38487eee-a666-4184-bf5a-a6555765f16b\" (UID: \"38487eee-a666-4184-bf5a-a6555765f16b\") "
Sep 16 21:43:22.051819 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.051714    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38487eee-a666-4184-bf5a-a6555765f16b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "38487eee-a666-4184-bf5a-a6555765f16b" (UID: "38487eee-a666-4184-bf5a-a6555765f16b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:43:22.054299 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.054268    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38487eee-a666-4184-bf5a-a6555765f16b-kube-api-access-tmpr9" (OuterVolumeSpecName: "kube-api-access-tmpr9") pod "38487eee-a666-4184-bf5a-a6555765f16b" (UID: "38487eee-a666-4184-bf5a-a6555765f16b"). InnerVolumeSpecName "kube-api-access-tmpr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:43:22.054424 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.054296    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "38487eee-a666-4184-bf5a-a6555765f16b" (UID: "38487eee-a666-4184-bf5a-a6555765f16b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:43:22.054658 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.054624    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "38487eee-a666-4184-bf5a-a6555765f16b" (UID: "38487eee-a666-4184-bf5a-a6555765f16b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:43:22.152475 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.152383    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38487eee-a666-4184-bf5a-a6555765f16b-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:43:22.152475 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.152418    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:43:22.152475 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.152428    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmpr9\" (UniqueName: \"kubernetes.io/projected/38487eee-a666-4184-bf5a-a6555765f16b-kube-api-access-tmpr9\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:43:22.152475 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.152437    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38487eee-a666-4184-bf5a-a6555765f16b-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:43:22.340242 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.340201    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:22.467038 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.466954    2567 generic.go:356] "Generic (PLEG): container finished" podID="38487eee-a666-4184-bf5a-a6555765f16b" containerID="4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa" exitCode=0
Sep 16 21:43:22.467038 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.466995    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"
Sep 16 21:43:22.467038 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.467024    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk" event={"ID":"38487eee-a666-4184-bf5a-a6555765f16b","Type":"ContainerDied","Data":"4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa"}
Sep 16 21:43:22.467285 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.467081    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk" event={"ID":"38487eee-a666-4184-bf5a-a6555765f16b","Type":"ContainerDied","Data":"78c62a810b4d87d5d362449c514cfbcab8be52c84b1481f97a8fcdb6818f1f8c"}
Sep 16 21:43:22.467285 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.467104    2567 scope.go:117] "RemoveContainer" containerID="4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa"
Sep 16 21:43:22.477030 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.476988    2567 scope.go:117] "RemoveContainer" containerID="4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa"
Sep 16 21:43:22.477279 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:43:22.477261    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa\": container with ID starting with 4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa not found: ID does not exist" containerID="4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa"
Sep 16 21:43:22.477333 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.477288    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa"} err="failed to get container status \"4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa\": rpc error: code = NotFound desc = could not find container \"4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa\": container with ID starting with 4b85d7b285ff44c92d77825ad14c1063a2e8ca9695bf780416d9f6992e3d73fa not found: ID does not exist"
Sep 16 21:43:22.493940 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.493911    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"]
Sep 16 21:43:22.499169 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:22.499146    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5ee44a89-6b5ff85797-xrhtk"]
Sep 16 21:43:23.164292 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:23.164253    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:23.952455 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:23.952422    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38487eee-a666-4184-bf5a-a6555765f16b" path="/var/lib/kubelet/pods/38487eee-a666-4184-bf5a-a6555765f16b/volumes"
Sep 16 21:43:24.163981 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:24.163938    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:24.926570 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:24.926534    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:25.099861 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:25.099830    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:25.484728 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:25.484691    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:25.737268 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:25.737164    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:26.459543 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:26.459481    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:27.164613 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:27.164582    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:30.173719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:30.173681    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:43:44.789676 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:43:44.789638    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:05.700685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:05.700638    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:06.973222 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:06.973191    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:07.102955 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:07.102883    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:12.722315 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:12.722277    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:12.786483 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:12.786444    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:12.988748 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:12.988669    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:14.218890 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:14.218857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:14.321769 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:14.321729    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:14.445207 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:14.445163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:15.069514 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:15.069484    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:15.169920 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:15.169863    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:15.270607 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:15.270553    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:16.568300 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:16.568214    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:22.396382 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:22.396341    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:22.496790 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:22.496758    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:23.076016 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:23.075977    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:25.442260 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:25.442220    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:26.294086 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:26.294046    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:29.211144 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:29.211109    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:29.279585 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:29.279540    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:29.377932 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:29.377874    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:29.479753 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:29.479637    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:30.824168 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:30.824136    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:39.944994 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:39.944936    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:44:40.172827 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:44:40.172794    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:01.125752 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:01.125688    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:06.783965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:06.783918    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:06.852038 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:06.852002    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:07.157196 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:07.157096    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:07.493009 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:07.492890    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:07.646490 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:07.646449    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:10.049177 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:10.049142    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:10.940007 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:10.939972    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:11.024445 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:11.024408    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:11.922531 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:11.922496    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:12.764719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:12.764684    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:13.697698 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:13.697666    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:14.335572 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:14.335533    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:15.096157 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:15.096118    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:15.990545 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:15.990509    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"]
Sep 16 21:45:15.990980 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:15.990964    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38487eee-a666-4184-bf5a-a6555765f16b" containerName="maas-api"
Sep 16 21:45:15.990980 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:15.990979    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="38487eee-a666-4184-bf5a-a6555765f16b" containerName="maas-api"
Sep 16 21:45:15.991774 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:15.991747    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="38487eee-a666-4184-bf5a-a6555765f16b" containerName="maas-api"
Sep 16 21:45:16.000225 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.000201    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.009954 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.006047    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"]
Sep 16 21:45:16.009954 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.006817    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0de41b-serving-cert\""
Sep 16 21:45:16.022538 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.022503    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:16.075059 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.075007    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-maas-api-tls\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.075227 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.075091    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b8d1-893f-419b-8add-272556059f17-openshift-service-ca\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.075227 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.075140    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46ppk\" (UniqueName: \"kubernetes.io/projected/ad02b8d1-893f-419b-8add-272556059f17-kube-api-access-46ppk\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.075227 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.075207    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-metrics-certs\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.089029 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.088980    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:16.175857 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.175817    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-maas-api-tls\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.176263 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.175865    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b8d1-893f-419b-8add-272556059f17-openshift-service-ca\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.176263 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.175941    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46ppk\" (UniqueName: \"kubernetes.io/projected/ad02b8d1-893f-419b-8add-272556059f17-kube-api-access-46ppk\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.176263 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.175976    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-metrics-certs\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.176622 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.176594    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b8d1-893f-419b-8add-272556059f17-openshift-service-ca\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.178587 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.178558    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-maas-api-tls\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.178708 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.178558    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-metrics-certs\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.184013 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.183989    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46ppk\" (UniqueName: \"kubernetes.io/projected/ad02b8d1-893f-419b-8add-272556059f17-kube-api-access-46ppk\") pod \"maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.185336 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.185309    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:16.317615 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.317579    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:16.455842 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.455803    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:16.548753 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.548698    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:16.592928 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.592867    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"]
Sep 16 21:45:16.594867 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:45:16.594835    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad02b8d1_893f_419b_8add_272556059f17.slice/crio-eedaa3111d6042aecb39b60ea644d330ad6c8466dd3ae89bef48c70cc2884f78 WatchSource:0}: Error finding container eedaa3111d6042aecb39b60ea644d330ad6c8466dd3ae89bef48c70cc2884f78: Status 404 returned error can't find the container with id eedaa3111d6042aecb39b60ea644d330ad6c8466dd3ae89bef48c70cc2884f78
Sep 16 21:45:16.596327 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:16.596309    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:45:17.087779 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:17.087744    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk" event={"ID":"ad02b8d1-893f-419b-8add-272556059f17","Type":"ContainerStarted","Data":"86c0c01d3f4ff083c7fb7f1bca4ac564535b3869d8fc45a96c92cddac6d38e70"}
Sep 16 21:45:17.087953 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:17.087801    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk" event={"ID":"ad02b8d1-893f-419b-8add-272556059f17","Type":"ContainerStarted","Data":"eedaa3111d6042aecb39b60ea644d330ad6c8466dd3ae89bef48c70cc2884f78"}
Sep 16 21:45:17.087953 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:17.087869    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:17.106368 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:17.106319    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk" podStartSLOduration=1.711679143 podStartE2EDuration="2.106302119s" totalImagesPullingTime="394.622976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:17.10442539 +0000 UTC m=+1403.752837379" watchObservedRunningTime="2026-09-16 21:45:17.106302119 +0000 UTC m=+1403.754714106"
Sep 16 21:45:23.098084 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:23.098049    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:45:38.031088 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:38.031043    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:38.833265 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:38.833226    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:39.239558 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:39.239457    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:39.841244 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:39.841195    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:40.641613 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:40.641573    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:40.763796 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:40.763760    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:41.223189 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:41.223152    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:41.337545 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:41.337515    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:41.520706 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:41.520675    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:41.835886 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:41.835798    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:42.136187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:42.136085    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:42.819957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:42.819919    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:43.564564 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:43.564529    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:44.225307 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:44.225262    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"]
Sep 16 21:45:44.239451 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:44.239418    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"]
Sep 16 21:45:44.303127 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:44.303097    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:44.378858 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:44.378826    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:44.927296 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:44.927258    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:45.033771 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:45.033731    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:45.135396 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:45.135363    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:45.408642 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:45.408607    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:46.422725 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.422649    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"]
Sep 16 21:45:46.426590 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.426568    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.429212 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.429188    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0102fa-serving-cert\""
Sep 16 21:45:46.441393 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.441372    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"]
Sep 16 21:45:46.464583 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.464548    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:46.567756 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.567713    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:46.576199 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.576177    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-maas-api-tls\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.576339 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.576227    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-metrics-certs\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.576339 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.576310    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-openshift-service-ca\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.576339 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.576336    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pcq\" (UniqueName: \"kubernetes.io/projected/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-kube-api-access-p7pcq\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.664919 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.664873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:46.677176 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.677093    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-openshift-service-ca\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.677176 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.677127    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pcq\" (UniqueName: \"kubernetes.io/projected/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-kube-api-access-p7pcq\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.677397 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.677267    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-maas-api-tls\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.677397 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.677357    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-metrics-certs\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.677788 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.677747    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-openshift-service-ca\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.679968 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.679937    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-maas-api-tls\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.679968 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.679935    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-metrics-certs\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.685498 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.685475    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pcq\" (UniqueName: \"kubernetes.io/projected/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-kube-api-access-p7pcq\") pod \"maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:46.741411 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:46.741378    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:47.024874 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:47.024833    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:47.030679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:47.030650    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"]
Sep 16 21:45:47.033936 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:45:47.033894    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ff8aba_9efc_4024_9dc2_40bfeb12b6f1.slice/crio-f774c27ea3b9351799ed75ed1785a83361b5316d357a6a0650c76cd1926c551d WatchSource:0}: Error finding container f774c27ea3b9351799ed75ed1785a83361b5316d357a6a0650c76cd1926c551d: Status 404 returned error can't find the container with id f774c27ea3b9351799ed75ed1785a83361b5316d357a6a0650c76cd1926c551d
Sep 16 21:45:47.121776 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:47.121738    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:47.258676 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:47.258642    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph" event={"ID":"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1","Type":"ContainerStarted","Data":"f774c27ea3b9351799ed75ed1785a83361b5316d357a6a0650c76cd1926c551d"}
Sep 16 21:45:47.540246 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:47.540216    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:48.226113 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:48.226070    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:48.266139 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:48.266100    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph" event={"ID":"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1","Type":"ContainerStarted","Data":"c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a"}
Sep 16 21:45:48.266341 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:48.266163    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:48.286332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:48.286276    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph" podStartSLOduration=1.84815624 podStartE2EDuration="2.286259677s" totalImagesPullingTime="438.103437ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:45:48.283805483 +0000 UTC m=+1434.932217472" watchObservedRunningTime="2026-09-16 21:45:48.286259677 +0000 UTC m=+1434.934671701"
Sep 16 21:45:48.859499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:48.859464    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:49.611692 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:49.611654    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:50.280065 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:50.280027    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:51.024755 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:51.024720    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:51.805339 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:51.805299    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:52.622328 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:52.622291    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:53.262664 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:53.262632    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:54.119367 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:54.119328    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:54.277767 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:54.277732    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:45:58.809919 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:58.809879    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"]
Sep 16 21:45:58.810371 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:58.810114    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q" podUID="220bf348-bef5-4886-8dcb-bd7ea4704222" containerName="maas-api" containerID="cri-o://6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32" gracePeriod=30
Sep 16 21:45:58.846951 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:58.846916    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:58.942771 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:58.942734    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:59.043966 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.043932    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:59.074485 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.074413    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:45:59.142956 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.142920    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:59.208491 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.208449    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-maas-api-tls\") pod \"220bf348-bef5-4886-8dcb-bd7ea4704222\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") "
Sep 16 21:45:59.208702 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.208568    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cncl\" (UniqueName: \"kubernetes.io/projected/220bf348-bef5-4886-8dcb-bd7ea4704222-kube-api-access-8cncl\") pod \"220bf348-bef5-4886-8dcb-bd7ea4704222\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") "
Sep 16 21:45:59.208702 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.208600    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bf348-bef5-4886-8dcb-bd7ea4704222-openshift-service-ca\") pod \"220bf348-bef5-4886-8dcb-bd7ea4704222\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") "
Sep 16 21:45:59.208702 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.208685    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-metrics-certs\") pod \"220bf348-bef5-4886-8dcb-bd7ea4704222\" (UID: \"220bf348-bef5-4886-8dcb-bd7ea4704222\") "
Sep 16 21:45:59.209353 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.209307    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220bf348-bef5-4886-8dcb-bd7ea4704222-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "220bf348-bef5-4886-8dcb-bd7ea4704222" (UID: "220bf348-bef5-4886-8dcb-bd7ea4704222"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:45:59.211788 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.211579    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/220bf348-bef5-4886-8dcb-bd7ea4704222-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.218804 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.218759    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "220bf348-bef5-4886-8dcb-bd7ea4704222" (UID: "220bf348-bef5-4886-8dcb-bd7ea4704222"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:45:59.219562 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.219382    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220bf348-bef5-4886-8dcb-bd7ea4704222-kube-api-access-8cncl" (OuterVolumeSpecName: "kube-api-access-8cncl") pod "220bf348-bef5-4886-8dcb-bd7ea4704222" (UID: "220bf348-bef5-4886-8dcb-bd7ea4704222"). InnerVolumeSpecName "kube-api-access-8cncl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:45:59.220831 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.220797    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "220bf348-bef5-4886-8dcb-bd7ea4704222" (UID: "220bf348-bef5-4886-8dcb-bd7ea4704222"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:45:59.249921 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.249870    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q" podUID="c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" containerName="main" containerID="cri-o://0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11" gracePeriod=45
Sep 16 21:45:59.260507 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.260401    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm" podUID="b4a426c3-1bd3-4311-8bab-a9a419e77e16" containerName="main" containerID="cri-o://a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669" gracePeriod=45
Sep 16 21:45:59.313113 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.313080    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cncl\" (UniqueName: \"kubernetes.io/projected/220bf348-bef5-4886-8dcb-bd7ea4704222-kube-api-access-8cncl\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.313113 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.313111    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.313113 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.313121    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/220bf348-bef5-4886-8dcb-bd7ea4704222-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.329190 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.329116    2567 generic.go:356] "Generic (PLEG): container finished" podID="220bf348-bef5-4886-8dcb-bd7ea4704222" containerID="6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32" exitCode=0
Sep 16 21:45:59.329190 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.329178    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"
Sep 16 21:45:59.329348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.329192    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q" event={"ID":"220bf348-bef5-4886-8dcb-bd7ea4704222","Type":"ContainerDied","Data":"6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32"}
Sep 16 21:45:59.329348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.329266    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q" event={"ID":"220bf348-bef5-4886-8dcb-bd7ea4704222","Type":"ContainerDied","Data":"b5242741ad6bdcf4ef1b9312d5beb713537ef0e76c4bd1241583d01c440d8913"}
Sep 16 21:45:59.329348 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.329289    2567 scope.go:117] "RemoveContainer" containerID="6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32"
Sep 16 21:45:59.361250 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.361219    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"]
Sep 16 21:45:59.365362 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.365334    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-35f5fd-648b58744b-ds55q"]
Sep 16 21:45:59.373370 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.373343    2567 scope.go:117] "RemoveContainer" containerID="6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32"
Sep 16 21:45:59.373747 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:45:59.373723    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32\": container with ID starting with 6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32 not found: ID does not exist" containerID="6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32"
Sep 16 21:45:59.373848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.373756    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32"} err="failed to get container status \"6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32\": rpc error: code = NotFound desc = could not find container \"6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32\": container with ID starting with 6fde9b1741f8cb48d6897a92c9715ac47907c4df6baa3800242a8d252a0e1d32 not found: ID does not exist"
Sep 16 21:45:59.403869 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.403824    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:59.506714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.506681    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:45:59.525185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.525158    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:45:59.536582 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.536561    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:45:59.716194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716118    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-model-cache\") pod \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") "
Sep 16 21:45:59.716194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716153    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k847p\" (UniqueName: \"kubernetes.io/projected/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-kube-api-access-k847p\") pod \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") "
Sep 16 21:45:59.716194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716191    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-dshm\") pod \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") "
Sep 16 21:45:59.716429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716210    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-home\") pod \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") "
Sep 16 21:45:59.716429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716231    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tls-certs\") pod \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") "
Sep 16 21:45:59.716429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716279    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tmp-dir\") pod \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") "
Sep 16 21:45:59.716429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716331    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tmp-dir\") pod \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") "
Sep 16 21:45:59.716429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716370    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-dshm\") pod \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") "
Sep 16 21:45:59.716429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716393    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-home\") pod \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") "
Sep 16 21:45:59.716429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716427    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-model-cache\") pod \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") "
Sep 16 21:45:59.716802 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716460    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5qd6\" (UniqueName: \"kubernetes.io/projected/b4a426c3-1bd3-4311-8bab-a9a419e77e16-kube-api-access-g5qd6\") pod \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\" (UID: \"b4a426c3-1bd3-4311-8bab-a9a419e77e16\") "
Sep 16 21:45:59.716802 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716467    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4a426c3-1bd3-4311-8bab-a9a419e77e16" (UID: "b4a426c3-1bd3-4311-8bab-a9a419e77e16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.716802 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716518    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-home" (OuterVolumeSpecName: "home") pod "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" (UID: "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.716802 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716545    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tls-certs\") pod \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\" (UID: \"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a\") "
Sep 16 21:45:59.716802 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.716765    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" (UID: "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.717087 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.717022    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4a426c3-1bd3-4311-8bab-a9a419e77e16" (UID: "b4a426c3-1bd3-4311-8bab-a9a419e77e16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.717087 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.717035    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-home" (OuterVolumeSpecName: "home") pod "b4a426c3-1bd3-4311-8bab-a9a419e77e16" (UID: "b4a426c3-1bd3-4311-8bab-a9a419e77e16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.717252 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.717230    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" (UID: "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.717373 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.717354    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-model-cache\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.717463 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.717378    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-home\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.717463 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.717393    2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-model-cache\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.718707 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.718679    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4a426c3-1bd3-4311-8bab-a9a419e77e16" (UID: "b4a426c3-1bd3-4311-8bab-a9a419e77e16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:45:59.718816 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.718741    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-kube-api-access-k847p" (OuterVolumeSpecName: "kube-api-access-k847p") pod "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" (UID: "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a"). InnerVolumeSpecName "kube-api-access-k847p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:45:59.718816 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.718807    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-dshm" (OuterVolumeSpecName: "dshm") pod "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" (UID: "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.718935 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.718828    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-dshm" (OuterVolumeSpecName: "dshm") pod "b4a426c3-1bd3-4311-8bab-a9a419e77e16" (UID: "b4a426c3-1bd3-4311-8bab-a9a419e77e16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 21:45:59.718935 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.718925    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" (UID: "c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:45:59.719164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.719148    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a426c3-1bd3-4311-8bab-a9a419e77e16-kube-api-access-g5qd6" (OuterVolumeSpecName: "kube-api-access-g5qd6") pod "b4a426c3-1bd3-4311-8bab-a9a419e77e16" (UID: "b4a426c3-1bd3-4311-8bab-a9a419e77e16"). InnerVolumeSpecName "kube-api-access-g5qd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:45:59.818957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.818923    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tmp-dir\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.818957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.818956    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-dshm\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.818957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.818968    2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-home\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.819402 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.818977    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5qd6\" (UniqueName: \"kubernetes.io/projected/b4a426c3-1bd3-4311-8bab-a9a419e77e16-kube-api-access-g5qd6\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.819402 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.818985    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-tls-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.819402 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.818993    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k847p\" (UniqueName: \"kubernetes.io/projected/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-kube-api-access-k847p\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.819402 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.819004    2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a-dshm\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.819402 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.819012    2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tls-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.819402 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.819019    2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4a426c3-1bd3-4311-8bab-a9a419e77e16-tmp-dir\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:45:59.943217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:45:59.943182    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="220bf348-bef5-4886-8dcb-bd7ea4704222" path="/var/lib/kubelet/pods/220bf348-bef5-4886-8dcb-bd7ea4704222/volumes"
Sep 16 21:46:00.213867 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.213837    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:00.336884 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.336849    2567 generic.go:356] "Generic (PLEG): container finished" podID="c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" containerID="0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11" exitCode=0
Sep 16 21:46:00.337078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.336893    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"
Sep 16 21:46:00.337078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.336936    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q" event={"ID":"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a","Type":"ContainerDied","Data":"0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11"}
Sep 16 21:46:00.337078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.336981    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q" event={"ID":"c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a","Type":"ContainerDied","Data":"2da74a107fb3d304de1c9df2254f6e637661089729ed20caf6924b5ef8cbe1f9"}
Sep 16 21:46:00.337078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.337005    2567 scope.go:117] "RemoveContainer" containerID="0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11"
Sep 16 21:46:00.338499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.338329    2567 generic.go:356] "Generic (PLEG): container finished" podID="b4a426c3-1bd3-4311-8bab-a9a419e77e16" containerID="a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669" exitCode=0
Sep 16 21:46:00.338499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.338357    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm" event={"ID":"b4a426c3-1bd3-4311-8bab-a9a419e77e16","Type":"ContainerDied","Data":"a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669"}
Sep 16 21:46:00.338499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.338390    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"
Sep 16 21:46:00.338499 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.338392    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm" event={"ID":"b4a426c3-1bd3-4311-8bab-a9a419e77e16","Type":"ContainerDied","Data":"0238b6dc6c99e05e19942961cd36b6c8e0fe7acf8cd905405940ee3decfb6ef3"}
Sep 16 21:46:00.345863 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.345845    2567 scope.go:117] "RemoveContainer" containerID="0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11"
Sep 16 21:46:00.346153 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:46:00.346136    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11\": container with ID starting with 0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11 not found: ID does not exist" containerID="0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11"
Sep 16 21:46:00.346219 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.346161    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11"} err="failed to get container status \"0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11\": rpc error: code = NotFound desc = could not find container \"0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11\": container with ID starting with 0e19f469ded6d9906604cbfd8b36d4dbec375abd9f0f92b6ad20816c4fe1ca11 not found: ID does not exist"
Sep 16 21:46:00.346219 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.346177    2567 scope.go:117] "RemoveContainer" containerID="a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669"
Sep 16 21:46:00.353839 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.353812    2567 scope.go:117] "RemoveContainer" containerID="a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669"
Sep 16 21:46:00.354174 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:46:00.354154    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669\": container with ID starting with a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669 not found: ID does not exist" containerID="a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669"
Sep 16 21:46:00.354247 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.354184    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669"} err="failed to get container status \"a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669\": rpc error: code = NotFound desc = could not find container \"a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669\": container with ID starting with a9eb8cdf61f1c9baddd410d81d73416642c66468c5d4084bc997fdebd3572669 not found: ID does not exist"
Sep 16 21:46:00.357289 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.357269    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"]
Sep 16 21:46:00.367824 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.367803    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/distinct-w0-35f5fd-kserve-56b4647bd8-9fs4q"]
Sep 16 21:46:00.383619 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.383598    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"]
Sep 16 21:46:00.389293 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:00.389275    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-35f5fd/premium-w0-35f5fd-kserve-6c5f9c6fd5-nz6tm"]
Sep 16 21:46:01.035873 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:01.035840    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:01.937507 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:01.937473    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:01.943610 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:01.943582    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a426c3-1bd3-4311-8bab-a9a419e77e16" path="/var/lib/kubelet/pods/b4a426c3-1bd3-4311-8bab-a9a419e77e16/volumes"
Sep 16 21:46:01.944012 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:01.943997    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" path="/var/lib/kubelet/pods/c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a/volumes"
Sep 16 21:46:02.729692 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:02.729656    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:03.433012 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:03.432970    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:04.152718 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:04.152683    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:04.672187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:04.672140    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:04.935137 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:04.935051    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:05.768919 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:05.768870    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:05.869101 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:05.869057    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:06.087481 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:06.087386    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:06.157761 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:06.157729    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:06.362692 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:06.362592    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:06.509018 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:06.508982    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:07.322135 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:07.322099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:08.092572 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:08.092527    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:08.152420 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:08.152389    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:08.231372 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:08.231331    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:09.031567 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:09.031530    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:09.749645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:09.749615    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:10.428798 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:10.428758    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:11.123666 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:11.123628    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:11.783887 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:11.783850    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:12.721885 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:12.721856    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:13.522172 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:13.522142    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:17.752374 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:17.752337    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:19.027610 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:19.027579    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:19.128299 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:19.128268    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:21.786274 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:21.786227    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:21.886879 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:21.886848    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:22.140146 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:22.140055    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:23.284311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.284275    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"]
Sep 16 21:46:23.284818 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.284487    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk" podUID="ad02b8d1-893f-419b-8add-272556059f17" containerName="maas-api" containerID="cri-o://86c0c01d3f4ff083c7fb7f1bca4ac564535b3869d8fc45a96c92cddac6d38e70" gracePeriod=30
Sep 16 21:46:23.321623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.321577    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:23.421935 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.421877    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:23.464759 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.464591    2567 generic.go:356] "Generic (PLEG): container finished" podID="ad02b8d1-893f-419b-8add-272556059f17" containerID="86c0c01d3f4ff083c7fb7f1bca4ac564535b3869d8fc45a96c92cddac6d38e70" exitCode=0
Sep 16 21:46:23.464759 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.464729    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk" event={"ID":"ad02b8d1-893f-419b-8add-272556059f17","Type":"ContainerDied","Data":"86c0c01d3f4ff083c7fb7f1bca4ac564535b3869d8fc45a96c92cddac6d38e70"}
Sep 16 21:46:23.512100 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.512064    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:23.543245 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.543176    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:46:23.574936 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.574884    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:23.656476 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.656439    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46ppk\" (UniqueName: \"kubernetes.io/projected/ad02b8d1-893f-419b-8add-272556059f17-kube-api-access-46ppk\") pod \"ad02b8d1-893f-419b-8add-272556059f17\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") "
Sep 16 21:46:23.656673 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.656501    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-metrics-certs\") pod \"ad02b8d1-893f-419b-8add-272556059f17\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") "
Sep 16 21:46:23.656673 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.656609    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-maas-api-tls\") pod \"ad02b8d1-893f-419b-8add-272556059f17\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") "
Sep 16 21:46:23.656673 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.656655    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b8d1-893f-419b-8add-272556059f17-openshift-service-ca\") pod \"ad02b8d1-893f-419b-8add-272556059f17\" (UID: \"ad02b8d1-893f-419b-8add-272556059f17\") "
Sep 16 21:46:23.657085 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.657048    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad02b8d1-893f-419b-8add-272556059f17-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ad02b8d1-893f-419b-8add-272556059f17" (UID: "ad02b8d1-893f-419b-8add-272556059f17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:23.659086 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.659051    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ad02b8d1-893f-419b-8add-272556059f17" (UID: "ad02b8d1-893f-419b-8add-272556059f17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:46:23.659321 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.659284    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad02b8d1-893f-419b-8add-272556059f17-kube-api-access-46ppk" (OuterVolumeSpecName: "kube-api-access-46ppk") pod "ad02b8d1-893f-419b-8add-272556059f17" (UID: "ad02b8d1-893f-419b-8add-272556059f17"). InnerVolumeSpecName "kube-api-access-46ppk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:23.659517 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.659494    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ad02b8d1-893f-419b-8add-272556059f17" (UID: "ad02b8d1-893f-419b-8add-272556059f17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:46:23.757772 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.757735    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:46:23.757772 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.757769    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad02b8d1-893f-419b-8add-272556059f17-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:46:23.758008 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.757783    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46ppk\" (UniqueName: \"kubernetes.io/projected/ad02b8d1-893f-419b-8add-272556059f17-kube-api-access-46ppk\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:46:23.758008 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.757795    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad02b8d1-893f-419b-8add-272556059f17-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:46:23.858993 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.858892    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:23.959731 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:23.959700    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:24.471560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:24.471526    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk" event={"ID":"ad02b8d1-893f-419b-8add-272556059f17","Type":"ContainerDied","Data":"eedaa3111d6042aecb39b60ea644d330ad6c8466dd3ae89bef48c70cc2884f78"}
Sep 16 21:46:24.472025 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:24.471583    2567 scope.go:117] "RemoveContainer" containerID="86c0c01d3f4ff083c7fb7f1bca4ac564535b3869d8fc45a96c92cddac6d38e70"
Sep 16 21:46:24.472025 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:24.471542    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"
Sep 16 21:46:24.490820 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:24.490785    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"]
Sep 16 21:46:24.495670 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:24.495646    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0de41b-7f994c8697-lwbnk"]
Sep 16 21:46:24.838482 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:24.838453    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:25.521776 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:25.521745    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:25.943589 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:25.943494    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad02b8d1-893f-419b-8add-272556059f17" path="/var/lib/kubelet/pods/ad02b8d1-893f-419b-8add-272556059f17/volumes"
Sep 16 21:46:26.332594 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:26.332554    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:27.116582 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:27.116545    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:27.758885 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:27.758847    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:28.451623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:28.451590    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:29.733467 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:29.733423    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:29.839533 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:29.839500    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:30.033823 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:30.033779    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:30.658369 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:30.658333    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:31.593034 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:31.592989    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:32.324954 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:32.324919    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:33.016665 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:33.016631    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:33.686258 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:33.686225    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:34.327477 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:34.327436    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:35.146145 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:35.146111    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:35.982563 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:35.982528    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:45.879106 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:45.879074    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"]
Sep 16 21:46:45.879653 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:45.879291    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph" podUID="73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" containerName="maas-api" containerID="cri-o://c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a" gracePeriod=30
Sep 16 21:46:46.133381 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.133312    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:46:46.175674 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.175645    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-metrics-certs\") pod \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") "
Sep 16 21:46:46.175862 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.175714    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-openshift-service-ca\") pod \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") "
Sep 16 21:46:46.175862 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.175763    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7pcq\" (UniqueName: \"kubernetes.io/projected/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-kube-api-access-p7pcq\") pod \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") "
Sep 16 21:46:46.175862 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.175800    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-maas-api-tls\") pod \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\" (UID: \"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1\") "
Sep 16 21:46:46.176181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.176137    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" (UID: "73ff8aba-9efc-4024-9dc2-40bfeb12b6f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:46:46.178046 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.178017    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" (UID: "73ff8aba-9efc-4024-9dc2-40bfeb12b6f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:46:46.178166 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.178064    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" (UID: "73ff8aba-9efc-4024-9dc2-40bfeb12b6f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:46:46.178166 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.178146    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-kube-api-access-p7pcq" (OuterVolumeSpecName: "kube-api-access-p7pcq") pod "73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" (UID: "73ff8aba-9efc-4024-9dc2-40bfeb12b6f1"). InnerVolumeSpecName "kube-api-access-p7pcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:46:46.277377 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.277337    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7pcq\" (UniqueName: \"kubernetes.io/projected/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-kube-api-access-p7pcq\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:46:46.277377 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.277374    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:46:46.277377 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.277383    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:46:46.277614 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.277391    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:46:46.592722 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.592688    2567 generic.go:356] "Generic (PLEG): container finished" podID="73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" containerID="c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a" exitCode=0
Sep 16 21:46:46.592965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.592739    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"
Sep 16 21:46:46.592965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.592752    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph" event={"ID":"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1","Type":"ContainerDied","Data":"c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a"}
Sep 16 21:46:46.592965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.592795    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph" event={"ID":"73ff8aba-9efc-4024-9dc2-40bfeb12b6f1","Type":"ContainerDied","Data":"f774c27ea3b9351799ed75ed1785a83361b5316d357a6a0650c76cd1926c551d"}
Sep 16 21:46:46.592965 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.592816    2567 scope.go:117] "RemoveContainer" containerID="c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a"
Sep 16 21:46:46.602589 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.602570    2567 scope.go:117] "RemoveContainer" containerID="c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a"
Sep 16 21:46:46.602874 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:46:46.602850    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a\": container with ID starting with c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a not found: ID does not exist" containerID="c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a"
Sep 16 21:46:46.603000 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.602881    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a"} err="failed to get container status \"c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a\": rpc error: code = NotFound desc = could not find container \"c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a\": container with ID starting with c4c934811ec8023f8a18263a9ea2cfe3a31599a2118e421b65f8cc401319a38a not found: ID does not exist"
Sep 16 21:46:46.635704 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.635666    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"]
Sep 16 21:46:46.643202 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.643166    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0102fa-7b56d85f54-rfxph"]
Sep 16 21:46:46.822274 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.822242    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:46.900518 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:46.900412    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:47.473849 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:47.473814    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:47.942100 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:47.942065    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" path="/var/lib/kubelet/pods/73ff8aba-9efc-4024-9dc2-40bfeb12b6f1/volumes"
Sep 16 21:46:48.142412 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:48.142375    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:48.920767 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:48.920732    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:49.597928 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:49.597864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:50.234685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:50.234644    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:52.973896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:52.973856    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:53.076604 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:53.076569    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:53.234227 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:53.234116    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:53.918845 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:53.918815    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:46:53.919106 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:53.919085    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:46:54.472050 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:54.472014    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:55.241552 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:55.241511    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:56.172678 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:56.172636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:56.852680 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:56.852649    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:57.606949 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:57.606894    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:46:58.293388 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:46:58.293342    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:04.106120 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.106084    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"]
Sep 16 21:47:04.106617 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.106330    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb" podUID="babb773c-f5e2-4c96-9b02-53db1eda5f6a" containerName="maas-api" containerID="cri-o://84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d" gracePeriod=30
Sep 16 21:47:04.129559 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.129535    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:04.205714 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.205673    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:04.304653 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.304596    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:04.362573 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.362496    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:47:04.386086 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.386046    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:04.457429 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.457397    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnflx\" (UniqueName: \"kubernetes.io/projected/babb773c-f5e2-4c96-9b02-53db1eda5f6a-kube-api-access-mnflx\") pod \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") "
Sep 16 21:47:04.457618 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.457470    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-metrics-certs\") pod \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") "
Sep 16 21:47:04.457618 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.457502    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/babb773c-f5e2-4c96-9b02-53db1eda5f6a-openshift-service-ca\") pod \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") "
Sep 16 21:47:04.457618 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.457538    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-maas-api-tls\") pod \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\" (UID: \"babb773c-f5e2-4c96-9b02-53db1eda5f6a\") "
Sep 16 21:47:04.457889 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.457854    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babb773c-f5e2-4c96-9b02-53db1eda5f6a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "babb773c-f5e2-4c96-9b02-53db1eda5f6a" (UID: "babb773c-f5e2-4c96-9b02-53db1eda5f6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:04.459749 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.459711    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "babb773c-f5e2-4c96-9b02-53db1eda5f6a" (UID: "babb773c-f5e2-4c96-9b02-53db1eda5f6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:04.459870 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.459759    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babb773c-f5e2-4c96-9b02-53db1eda5f6a-kube-api-access-mnflx" (OuterVolumeSpecName: "kube-api-access-mnflx") pod "babb773c-f5e2-4c96-9b02-53db1eda5f6a" (UID: "babb773c-f5e2-4c96-9b02-53db1eda5f6a"). InnerVolumeSpecName "kube-api-access-mnflx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:04.459870 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.459798    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "babb773c-f5e2-4c96-9b02-53db1eda5f6a" (UID: "babb773c-f5e2-4c96-9b02-53db1eda5f6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:04.559124 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.559088    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:04.559124 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.559126    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnflx\" (UniqueName: \"kubernetes.io/projected/babb773c-f5e2-4c96-9b02-53db1eda5f6a-kube-api-access-mnflx\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:04.559317 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.559140    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/babb773c-f5e2-4c96-9b02-53db1eda5f6a-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:04.559317 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.559154    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/babb773c-f5e2-4c96-9b02-53db1eda5f6a-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:04.689549 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.689453    2567 generic.go:356] "Generic (PLEG): container finished" podID="babb773c-f5e2-4c96-9b02-53db1eda5f6a" containerID="84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d" exitCode=0
Sep 16 21:47:04.689549 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.689504    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"
Sep 16 21:47:04.689800 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.689547    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb" event={"ID":"babb773c-f5e2-4c96-9b02-53db1eda5f6a","Type":"ContainerDied","Data":"84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d"}
Sep 16 21:47:04.689800 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.689617    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb" event={"ID":"babb773c-f5e2-4c96-9b02-53db1eda5f6a","Type":"ContainerDied","Data":"ae29390e7a649aa9b8a360326a2ad61a0b2e00d785f7182345551d98130e858d"}
Sep 16 21:47:04.689800 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.689647    2567 scope.go:117] "RemoveContainer" containerID="84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d"
Sep 16 21:47:04.699957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.699929    2567 scope.go:117] "RemoveContainer" containerID="84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d"
Sep 16 21:47:04.700291 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:47:04.700272    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d\": container with ID starting with 84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d not found: ID does not exist" containerID="84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d"
Sep 16 21:47:04.700377 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.700298    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d"} err="failed to get container status \"84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d\": rpc error: code = NotFound desc = could not find container \"84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d\": container with ID starting with 84039db2e50562d1a0cfc01d906251edea434a6202d21f839e1a01161012a77d not found: ID does not exist"
Sep 16 21:47:04.710755 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.710724    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"]
Sep 16 21:47:04.714313 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.714289    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-296002-68fb964465-8qwwb"]
Sep 16 21:47:04.724911 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.724864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:04.783827 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:04.783789    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:05.389109 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:05.389073    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:05.943042 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:05.943006    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babb773c-f5e2-4c96-9b02-53db1eda5f6a" path="/var/lib/kubelet/pods/babb773c-f5e2-4c96-9b02-53db1eda5f6a/volumes"
Sep 16 21:47:06.038727 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:06.038693    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:06.839397 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:06.839363    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:07.480685 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:07.480647    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:10.637851 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:10.637817    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:10.738257 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:10.738222    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:10.914701 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:10.914605    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:11.537708 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:11.537671    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:12.523652 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:12.523612    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:13.257453 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:13.257418    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:14.019398 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:14.019357    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:14.706805 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:14.706766    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:15.380276 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:15.380238    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:15.542141 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:15.542107    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:16.427644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:16.427565    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:17.239082 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:17.239049    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:17.937068 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:17.937031    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:18.619139 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:18.619109    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:19.265962 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:19.265928    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:20.638165 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:20.638131    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:21.306695 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:21.306655    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:21.803187 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:21.803158    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"]
Sep 16 21:47:21.803670 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:21.803371    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp" podUID="0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" containerName="maas-api" containerID="cri-o://55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f" gracePeriod=30
Sep 16 21:47:21.833736 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:21.833704    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:21.907789 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:21.907757    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:21.970645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:21.970602    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:22.058766 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.058695    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:47:22.129808 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.129775    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-metrics-certs\") pod \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") "
Sep 16 21:47:22.130005 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.129833    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls\") pod \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") "
Sep 16 21:47:22.130005 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.129864    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-openshift-service-ca\") pod \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") "
Sep 16 21:47:22.130005 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.129965    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfqrq\" (UniqueName: \"kubernetes.io/projected/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-kube-api-access-mfqrq\") pod \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\" (UID: \"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2\") "
Sep 16 21:47:22.130211 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.130182    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" (UID: "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:22.130341 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.130325    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:22.132020 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.131994    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-kube-api-access-mfqrq" (OuterVolumeSpecName: "kube-api-access-mfqrq") pod "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" (UID: "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2"). InnerVolumeSpecName "kube-api-access-mfqrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:22.132125 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.132017    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" (UID: "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:22.132125 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.132063    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" (UID: "0a4f5ad9-50ac-4d63-80e3-3e474a1040b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:22.227291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.227245    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:22.231573 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.231548    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:22.231573 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.231571    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:22.231753 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.231581    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfqrq\" (UniqueName: \"kubernetes.io/projected/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2-kube-api-access-mfqrq\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:22.333065 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.332971    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:22.431917 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.431874    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:22.532334 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.532305    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:22.790691 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.790660    2567 generic.go:356] "Generic (PLEG): container finished" podID="0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" containerID="55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f" exitCode=0
Sep 16 21:47:22.790893 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.790709    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"
Sep 16 21:47:22.790893 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.790741    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp" event={"ID":"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2","Type":"ContainerDied","Data":"55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f"}
Sep 16 21:47:22.790893 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.790804    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp" event={"ID":"0a4f5ad9-50ac-4d63-80e3-3e474a1040b2","Type":"ContainerDied","Data":"c94da0ca09567cb41ef53d74118b7801ca9a0aeca291fd7ad3a0b25ff60194b0"}
Sep 16 21:47:22.790893 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.790828    2567 scope.go:117] "RemoveContainer" containerID="55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f"
Sep 16 21:47:22.808956 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.808694    2567 scope.go:117] "RemoveContainer" containerID="55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f"
Sep 16 21:47:22.809234 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:47:22.809065    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f\": container with ID starting with 55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f not found: ID does not exist" containerID="55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f"
Sep 16 21:47:22.809234 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.809096    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f"} err="failed to get container status \"55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f\": rpc error: code = NotFound desc = could not find container \"55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f\": container with ID starting with 55e5844f8e584c2c0e685744ce5481ca4bbc61930c43a380f390cbec65109b3f not found: ID does not exist"
Sep 16 21:47:22.817778 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.817753    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"]
Sep 16 21:47:22.821647 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:22.821624    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-365ea1-7bb8d7fd7d-xm2rp"]
Sep 16 21:47:23.099932 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:23.099835    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:23.821108 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:23.821072    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:23.942628 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:23.942594    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" path="/var/lib/kubelet/pods/0a4f5ad9-50ac-4d63-80e3-3e474a1040b2/volumes"
Sep 16 21:47:24.634261 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:24.634220    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:28.114455 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:28.114407    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:28.214147 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:28.214113    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:28.389198 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:28.389103    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:31.154220 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.154188    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"]
Sep 16 21:47:31.154688 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.154430    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb" podUID="b453d27e-231b-4e5f-a1ff-219cfd63b1a2" containerName="maas-api" containerID="cri-o://551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2" gracePeriod=30
Sep 16 21:47:31.175004 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.174974    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:31.215039 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.215007    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"]
Sep 16 21:47:31.215258 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.215212    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v" podUID="a2c95c3c-01a9-434e-81c6-b2c8706d39ee" containerName="payload-processing" containerID="cri-o://13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170" gracePeriod=30
Sep 16 21:47:31.467756 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.467729    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:47:31.506852 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.506827    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:47:31.525692 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.525658    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-metrics-certs\") pod \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") "
Sep 16 21:47:31.525854 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.525745    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78rh7\" (UniqueName: \"kubernetes.io/projected/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-kube-api-access-78rh7\") pod \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") "
Sep 16 21:47:31.525854 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.525808    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-openshift-service-ca\") pod \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") "
Sep 16 21:47:31.525854 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.525835    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-maas-api-tls\") pod \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\" (UID: \"b453d27e-231b-4e5f-a1ff-219cfd63b1a2\") "
Sep 16 21:47:31.526314 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.526265    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b453d27e-231b-4e5f-a1ff-219cfd63b1a2" (UID: "b453d27e-231b-4e5f-a1ff-219cfd63b1a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:31.528558 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.528516    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b453d27e-231b-4e5f-a1ff-219cfd63b1a2" (UID: "b453d27e-231b-4e5f-a1ff-219cfd63b1a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:31.528558 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.528531    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-kube-api-access-78rh7" (OuterVolumeSpecName: "kube-api-access-78rh7") pod "b453d27e-231b-4e5f-a1ff-219cfd63b1a2" (UID: "b453d27e-231b-4e5f-a1ff-219cfd63b1a2"). InnerVolumeSpecName "kube-api-access-78rh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:31.528695 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.528590    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b453d27e-231b-4e5f-a1ff-219cfd63b1a2" (UID: "b453d27e-231b-4e5f-a1ff-219cfd63b1a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:47:31.627258 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.627212    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nstmj\" (UniqueName: \"kubernetes.io/projected/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-kube-api-access-nstmj\") pod \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\" (UID: \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\") "
Sep 16 21:47:31.627443 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.627307    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-plugins-config-volume\") pod \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\" (UID: \"a2c95c3c-01a9-434e-81c6-b2c8706d39ee\") "
Sep 16 21:47:31.627623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.627606    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:31.627660 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.627630    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78rh7\" (UniqueName: \"kubernetes.io/projected/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-kube-api-access-78rh7\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:31.627660 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.627645    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:31.627660 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.627657    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b453d27e-231b-4e5f-a1ff-219cfd63b1a2-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:31.627756 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.627650    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2c95c3c-01a9-434e-81c6-b2c8706d39ee" (UID: "a2c95c3c-01a9-434e-81c6-b2c8706d39ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:47:31.629512 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.629464    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-kube-api-access-nstmj" (OuterVolumeSpecName: "kube-api-access-nstmj") pod "a2c95c3c-01a9-434e-81c6-b2c8706d39ee" (UID: "a2c95c3c-01a9-434e-81c6-b2c8706d39ee"). InnerVolumeSpecName "kube-api-access-nstmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:47:31.728769 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.728726    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-plugins-config-volume\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:31.728769 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.728760    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nstmj\" (UniqueName: \"kubernetes.io/projected/a2c95c3c-01a9-434e-81c6-b2c8706d39ee-kube-api-access-nstmj\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:47:31.840344 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.840307    2567 generic.go:356] "Generic (PLEG): container finished" podID="a2c95c3c-01a9-434e-81c6-b2c8706d39ee" containerID="13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170" exitCode=0
Sep 16 21:47:31.840536 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.840360    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"
Sep 16 21:47:31.840536 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.840394    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v" event={"ID":"a2c95c3c-01a9-434e-81c6-b2c8706d39ee","Type":"ContainerDied","Data":"13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170"}
Sep 16 21:47:31.840536 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.840454    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v" event={"ID":"a2c95c3c-01a9-434e-81c6-b2c8706d39ee","Type":"ContainerDied","Data":"2edb0d8a16415ea7d27e7a5c8b54f2d2e96fa4f6c5e510d43439f1b7a7108878"}
Sep 16 21:47:31.840536 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.840487    2567 scope.go:117] "RemoveContainer" containerID="13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170"
Sep 16 21:47:31.841863 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.841842    2567 generic.go:356] "Generic (PLEG): container finished" podID="b453d27e-231b-4e5f-a1ff-219cfd63b1a2" containerID="551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2" exitCode=0
Sep 16 21:47:31.842013 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.841889    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"
Sep 16 21:47:31.842013 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.841947    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb" event={"ID":"b453d27e-231b-4e5f-a1ff-219cfd63b1a2","Type":"ContainerDied","Data":"551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2"}
Sep 16 21:47:31.842013 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.842002    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb" event={"ID":"b453d27e-231b-4e5f-a1ff-219cfd63b1a2","Type":"ContainerDied","Data":"864968d5b253c234b4053aa0c0684b9230a187e9a809e5c5f9617a812c02bbf5"}
Sep 16 21:47:31.850319 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.849969    2567 scope.go:117] "RemoveContainer" containerID="13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170"
Sep 16 21:47:31.850400 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:47:31.850329    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170\": container with ID starting with 13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170 not found: ID does not exist" containerID="13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170"
Sep 16 21:47:31.850400 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.850362    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170"} err="failed to get container status \"13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170\": rpc error: code = NotFound desc = could not find container \"13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170\": container with ID starting with 13b03a73d13510305940cd4da33f38009ebca0748ad6efee568cf520d467f170 not found: ID does not exist"
Sep 16 21:47:31.850400 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.850383    2567 scope.go:117] "RemoveContainer" containerID="551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2"
Sep 16 21:47:31.859056 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.859038    2567 scope.go:117] "RemoveContainer" containerID="551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2"
Sep 16 21:47:31.859316 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:47:31.859299    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2\": container with ID starting with 551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2 not found: ID does not exist" containerID="551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2"
Sep 16 21:47:31.859383 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.859322    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2"} err="failed to get container status \"551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2\": rpc error: code = NotFound desc = could not find container \"551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2\": container with ID starting with 551704d0125ac6abb192d4e64d52a59d433427de3155b33bb53a97dda468d4a2 not found: ID does not exist"
Sep 16 21:47:31.866402 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.866370    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"]
Sep 16 21:47:31.870333 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.870310    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-e047d8-7d788748df-whk7v"]
Sep 16 21:47:31.879844 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.879750    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"]
Sep 16 21:47:31.885703 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.885670    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb"]
Sep 16 21:47:31.943186 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.943151    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c95c3c-01a9-434e-81c6-b2c8706d39ee" path="/var/lib/kubelet/pods/a2c95c3c-01a9-434e-81c6-b2c8706d39ee/volumes"
Sep 16 21:47:31.943558 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:31.943545    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b453d27e-231b-4e5f-a1ff-219cfd63b1a2" path="/var/lib/kubelet/pods/b453d27e-231b-4e5f-a1ff-219cfd63b1a2/volumes"
Sep 16 21:47:32.232217 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:32.232181    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:32.792075 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:32.792041    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:33.484844 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:33.484799    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:34.432713 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:34.432664    2567 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-e047d8-6576dbd4b4-hzjnb" podUID="b453d27e-231b-4e5f-a1ff-219cfd63b1a2" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 21:47:35.014843 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:35.014764    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:35.087656 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:35.087607    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:37.821794 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:37.821757    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:37.882527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:37.882490    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:38.064366 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:38.064329    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:51.694545 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:51.694509    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:52.988537 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:52.988503    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:57.216401 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:57.216368    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:57.272462 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:57.272425    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:47:58.044822 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:47:58.044779    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:48:23.125468 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:48:23.125425    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:48:27.939301 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:48:27.939267    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:48:43.903915 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:48:43.903864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:48:54.680753 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:48:54.680710    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:48:54.738124 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:48:54.738083    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:49:16.719857 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:49:16.719823    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:49:17.691161 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:49:17.691130    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:49:24.735648 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:49:24.735605    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:49:46.567855 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:49:46.567820    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:49:48.954783 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:49:48.954747    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:50:09.312041 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:50:09.312002    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:50:54.929274 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:50:54.929237    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:04.953091 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:04.953053    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:06.015291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:06.015262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:06.433956 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:06.433843    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:06.725851 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:06.725778    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:07.033134 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:07.033106    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:09.447783 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:09.447751    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:09.672856 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:09.672819    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:09.766706 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:09.766676    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:10.284517 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:10.284485    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:11.114298 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.114262    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"]
Sep 16 21:51:11.114889 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.114867    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b453d27e-231b-4e5f-a1ff-219cfd63b1a2" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115072    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b453d27e-231b-4e5f-a1ff-219cfd63b1a2" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115103    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="220bf348-bef5-4886-8dcb-bd7ea4704222" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115114    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="220bf348-bef5-4886-8dcb-bd7ea4704222" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115124    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babb773c-f5e2-4c96-9b02-53db1eda5f6a" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115133    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="babb773c-f5e2-4c96-9b02-53db1eda5f6a" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115156    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad02b8d1-893f-419b-8add-272556059f17" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115164    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad02b8d1-893f-419b-8add-272556059f17" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115183    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c95c3c-01a9-434e-81c6-b2c8706d39ee" containerName="payload-processing"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115192    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c95c3c-01a9-434e-81c6-b2c8706d39ee" containerName="payload-processing"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115203    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a426c3-1bd3-4311-8bab-a9a419e77e16" containerName="main"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115210    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a426c3-1bd3-4311-8bab-a9a419e77e16" containerName="main"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115219    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115227    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115238    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115246    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115260    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" containerName="main"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115267    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" containerName="main"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115373    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a4f5ad9-50ac-4d63-80e3-3e474a1040b2" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115385    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="73ff8aba-9efc-4024-9dc2-40bfeb12b6f1" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115395    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a426c3-1bd3-4311-8bab-a9a419e77e16" containerName="main"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115404    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c54f2f1d-ea4f-4917-bd0b-fd0bbf94f89a" containerName="main"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115413    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c95c3c-01a9-434e-81c6-b2c8706d39ee" containerName="payload-processing"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115425    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b453d27e-231b-4e5f-a1ff-219cfd63b1a2" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115434    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="babb773c-f5e2-4c96-9b02-53db1eda5f6a" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115445    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="220bf348-bef5-4886-8dcb-bd7ea4704222" containerName="maas-api"
Sep 16 21:51:11.115784 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.115453    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad02b8d1-893f-419b-8add-272556059f17" containerName="maas-api"
Sep 16 21:51:11.119007 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.118987    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.121367 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.121347    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ad0222-serving-cert\""
Sep 16 21:51:11.129839 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.129814    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"]
Sep 16 21:51:11.141867 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.141833    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:11.176919 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.176871    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-metrics-certs\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.177114 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.176938    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pq4p\" (UniqueName: \"kubernetes.io/projected/203ee882-9cae-4884-8e75-3789ea89e023-kube-api-access-5pq4p\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.177114 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.177004    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.177114 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.177076    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ee882-9cae-4884-8e75-3789ea89e023-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.248826 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.248790    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:11.278505 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.278477    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-metrics-certs\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.278652 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.278518    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pq4p\" (UniqueName: \"kubernetes.io/projected/203ee882-9cae-4884-8e75-3789ea89e023-kube-api-access-5pq4p\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.278709 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.278688    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.278772 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.278757    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ee882-9cae-4884-8e75-3789ea89e023-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.279371 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.279347    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ee882-9cae-4884-8e75-3789ea89e023-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.280864 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.280843    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-metrics-certs\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.281030 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.281011    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.295383 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.295364    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pq4p\" (UniqueName: \"kubernetes.io/projected/203ee882-9cae-4884-8e75-3789ea89e023-kube-api-access-5pq4p\") pod \"maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.432176 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.432090    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:11.706172 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.705669    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"]
Sep 16 21:51:11.709846 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:51:11.709815    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203ee882_9cae_4884_8e75_3789ea89e023.slice/crio-8c83fd09a54f5e93218c82675a79b4e61432bd2a0aac2b17771dd841523e7252 WatchSource:0}: Error finding container 8c83fd09a54f5e93218c82675a79b4e61432bd2a0aac2b17771dd841523e7252: Status 404 returned error can't find the container with id 8c83fd09a54f5e93218c82675a79b4e61432bd2a0aac2b17771dd841523e7252
Sep 16 21:51:11.711516 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.711496    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:51:11.807423 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.807388    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:11.877742 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.877706    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:11.942930 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:11.942878    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28" event={"ID":"203ee882-9cae-4884-8e75-3789ea89e023","Type":"ContainerStarted","Data":"8c83fd09a54f5e93218c82675a79b4e61432bd2a0aac2b17771dd841523e7252"}
Sep 16 21:51:12.944645 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:12.944611    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28" event={"ID":"203ee882-9cae-4884-8e75-3789ea89e023","Type":"ContainerStarted","Data":"f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25"}
Sep 16 21:51:12.945075 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:12.944661    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:12.961379 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:12.961332    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28" podStartSLOduration=1.343962454 podStartE2EDuration="1.961315559s" totalImagesPullingTime="617.353105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:51:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:51:12.959245762 +0000 UTC m=+1759.607657750" watchObservedRunningTime="2026-09-16 21:51:12.961315559 +0000 UTC m=+1759.609727546"
Sep 16 21:51:18.954315 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:18.954284    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:51:26.224342 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:26.224304    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:26.279252 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:26.279220    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:28.445413 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:28.445373    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:29.218719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:29.218688    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:33.231508 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:33.231470    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:34.518792 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:34.518752    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:34.654247 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:34.654207    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:35.226496 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:35.226457    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:35.326976 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:35.326941    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:35.619433 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:35.619371    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:35.888725 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:35.888636    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:36.010912 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:36.010865    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:36.671017 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:36.670987    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:38.120946 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:38.120886    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:38.975309 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:38.975269    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:39.116459 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:39.116428    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:40.171026 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:40.170982    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:41.050750 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.050710    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:41.708078 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.708041    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"]
Sep 16 21:51:41.712407 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.712382    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.714594 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.714572    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-03eca6-serving-cert\""
Sep 16 21:51:41.723194 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.723168    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"]
Sep 16 21:51:41.738623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.738599    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:41.840317 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.840282    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:41.876959 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.876897    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.877129 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.876964    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w628x\" (UniqueName: \"kubernetes.io/projected/0e99efab-94a3-4a94-bda5-ae76d7c22be8-kube-api-access-w628x\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.877129 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.877025    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-metrics-certs\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.877214 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.877140    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e99efab-94a3-4a94-bda5-ae76d7c22be8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.940138 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.940100    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:41.978465 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.978373    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-metrics-certs\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.978465 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.978455    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e99efab-94a3-4a94-bda5-ae76d7c22be8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.978655 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.978486    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.978655 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.978507    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w628x\" (UniqueName: \"kubernetes.io/projected/0e99efab-94a3-4a94-bda5-ae76d7c22be8-kube-api-access-w628x\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.979209 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.979182    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e99efab-94a3-4a94-bda5-ae76d7c22be8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.980896 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.980875    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.981049 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.981031    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-metrics-certs\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:41.990142 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:41.990122    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w628x\" (UniqueName: \"kubernetes.io/projected/0e99efab-94a3-4a94-bda5-ae76d7c22be8-kube-api-access-w628x\") pod \"maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:42.025076 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:42.025040    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:42.220098 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:42.220065    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:42.302374 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:42.302334    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"]
Sep 16 21:51:42.304819 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:51:42.304791    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e99efab_94a3_4a94_bda5_ae76d7c22be8.slice/crio-3338e3f85d926b9165308bb4a0b66c9edf41ad20e99803a97428c601202b986d WatchSource:0}: Error finding container 3338e3f85d926b9165308bb4a0b66c9edf41ad20e99803a97428c601202b986d: Status 404 returned error can't find the container with id 3338e3f85d926b9165308bb4a0b66c9edf41ad20e99803a97428c601202b986d
Sep 16 21:51:42.389574 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:42.389538    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:51:43.099261 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:43.099223    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt" event={"ID":"0e99efab-94a3-4a94-bda5-ae76d7c22be8","Type":"ContainerStarted","Data":"ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069"}
Sep 16 21:51:43.099681 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:43.099280    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt" event={"ID":"0e99efab-94a3-4a94-bda5-ae76d7c22be8","Type":"ContainerStarted","Data":"3338e3f85d926b9165308bb4a0b66c9edf41ad20e99803a97428c601202b986d"}
Sep 16 21:51:43.099681 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:43.099395    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:43.117559 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:43.117505    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt" podStartSLOduration=1.7775929910000001 podStartE2EDuration="2.11749015s" totalImagesPullingTime="339.897159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:51:43.116253858 +0000 UTC m=+1789.764665872" watchObservedRunningTime="2026-09-16 21:51:43.11749015 +0000 UTC m=+1789.765902139"
Sep 16 21:51:49.110854 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:49.110765    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:51:53.952464 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:53.952436    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:51:53.955347 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:51:53.955323    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:52:03.737839 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:03.737808    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:03.837588 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:03.837540    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:04.644088 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:04.644049    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:05.624681 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:05.624640    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:06.313619 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:06.313579    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:06.982520 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:06.982483    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:22.207518 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:22.207484    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:43.399433 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:43.399392    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:44.563350 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:44.563314    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:45.482596 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:45.482558    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:45.778583 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:45.778539    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:46.050604 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:46.050520    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:49.027848 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:49.027808    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:49.235042 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:49.235013    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:49.319084 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:49.318999    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:49.923857 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:49.923827    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:50.758171 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:50.758119    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:51.447613 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:51.447561    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:52.409234 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.409201    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"]
Sep 16 21:52:52.413541 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.413517    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.416164 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.416141    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-dcbfbc-serving-cert\""
Sep 16 21:52:52.420912 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.420853    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"]
Sep 16 21:52:52.444353 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.444313    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:52.538063 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.538026    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.538287 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.538092    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6hd\" (UniqueName: \"kubernetes.io/projected/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-kube-api-access-bq6hd\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.538287 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.538133    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.538287 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.538217    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.545696 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.545663    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:52.639199 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.639167    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.639400 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.639223    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.639400 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.639270    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6hd\" (UniqueName: \"kubernetes.io/projected/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-kube-api-access-bq6hd\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.639400 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.639296    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.639400 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:52:52.639314    2567 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-serving-cert: secret "maas-api-e2e-ipp-cleanup-dcbfbc-serving-cert" not found
Sep 16 21:52:52.639400 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:52:52.639373    2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-maas-api-tls podName:f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a nodeName:}" failed. No retries permitted until 2026-09-16 21:52:53.139358675 +0000 UTC m=+1859.787770640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw" (UID: "f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a") : secret "maas-api-e2e-ipp-cleanup-dcbfbc-serving-cert" not found
Sep 16 21:52:52.640138 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.640117    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.641867 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.641846    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.650591 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.650566    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6hd\" (UniqueName: \"kubernetes.io/projected/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-kube-api-access-bq6hd\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:52.932950 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:52.932885    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:53.028869 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:53.028829    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:53.144583 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:53.144551    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:53.147050 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:53.147021    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:53.151008 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:53.150977    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:52:53.327265 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:53.327225    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:53.601727 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:53.601550    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"]
Sep 16 21:52:53.604462 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:52:53.604436    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b6184e_b5fc_4ba9_9384_e5ba8e1adc7a.slice/crio-8d8ba50ba9a03670253c6515e870ab74044ae1d01487609181391725108a9982 WatchSource:0}: Error finding container 8d8ba50ba9a03670253c6515e870ab74044ae1d01487609181391725108a9982: Status 404 returned error can't find the container with id 8d8ba50ba9a03670253c6515e870ab74044ae1d01487609181391725108a9982
Sep 16 21:52:54.458644 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:54.458609    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw" event={"ID":"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a","Type":"ContainerStarted","Data":"f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb"}
Sep 16 21:52:54.458838 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:54.458661    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw" event={"ID":"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a","Type":"ContainerStarted","Data":"8d8ba50ba9a03670253c6515e870ab74044ae1d01487609181391725108a9982"}
Sep 16 21:52:54.458838 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:54.458760    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:52:54.475746 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:52:54.475698    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw" podStartSLOduration=2.137514374 podStartE2EDuration="2.47568171s" totalImagesPullingTime="338.167336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:52:54.474532146 +0000 UTC m=+1861.122944136" watchObservedRunningTime="2026-09-16 21:52:54.47568171 +0000 UTC m=+1861.124093701"
Sep 16 21:53:00.468065 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:00.468034    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:53:15.456828 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:15.456788    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:16.261403 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:16.261310    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:16.927223 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:16.927185    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:17.726680 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:17.726648    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:18.445913 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:18.445863    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:19.203357 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:19.203319    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:19.332510 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:19.332481    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:20.925469 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:20.925426    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:21.636818 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:21.636778    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:21.737434 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:21.737403    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:22.398915 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:22.398872    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:23.095978 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:23.095873    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:23.764894 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:23.764846    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:24.633419 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:24.633383    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:30.845309 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:30.845270    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"]
Sep 16 21:53:30.845783 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:30.845580    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw" podUID="f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" containerName="maas-api" containerID="cri-o://f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb" gracePeriod=30
Sep 16 21:53:30.876719 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:30.876689    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:30.947836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:30.947804    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:31.011843 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.011640    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:31.075290 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.075255    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:31.102534 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.102468    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:53:31.139677 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.139641    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:31.203510 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.203477    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-maas-api-tls\") pod \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") "
Sep 16 21:53:31.203510 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.203517    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-metrics-certs\") pod \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") "
Sep 16 21:53:31.203779 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.203568    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-openshift-service-ca\") pod \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") "
Sep 16 21:53:31.203779 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.203668    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bq6hd\" (UniqueName: \"kubernetes.io/projected/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-kube-api-access-bq6hd\") pod \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\" (UID: \"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a\") "
Sep 16 21:53:31.204110 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.204080    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" (UID: "f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:53:31.205791 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.205770    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-kube-api-access-bq6hd" (OuterVolumeSpecName: "kube-api-access-bq6hd") pod "f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" (UID: "f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a"). InnerVolumeSpecName "kube-api-access-bq6hd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:53:31.206203 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.206178    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" (UID: "f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:53:31.206293 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.206186    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" (UID: "f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:53:31.304988 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.304947    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq6hd\" (UniqueName: \"kubernetes.io/projected/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-kube-api-access-bq6hd\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:53:31.304988 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.304979    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:53:31.304988 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.304988    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:53:31.304988 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.304996    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:53:31.431241 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.431127    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:31.535266 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.535230    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:31.644051 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.644016    2567 generic.go:356] "Generic (PLEG): container finished" podID="f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" containerID="f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb" exitCode=0
Sep 16 21:53:31.644221 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.644066    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"
Sep 16 21:53:31.644221 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.644105    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw" event={"ID":"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a","Type":"ContainerDied","Data":"f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb"}
Sep 16 21:53:31.644221 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.644154    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw" event={"ID":"f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a","Type":"ContainerDied","Data":"8d8ba50ba9a03670253c6515e870ab74044ae1d01487609181391725108a9982"}
Sep 16 21:53:31.644221 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.644181    2567 scope.go:117] "RemoveContainer" containerID="f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb"
Sep 16 21:53:31.655051 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.655027    2567 scope.go:117] "RemoveContainer" containerID="f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb"
Sep 16 21:53:31.655315 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:53:31.655297    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb\": container with ID starting with f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb not found: ID does not exist" containerID="f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb"
Sep 16 21:53:31.655365 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.655323    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb"} err="failed to get container status \"f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb\": rpc error: code = NotFound desc = could not find container \"f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb\": container with ID starting with f59e07d1ff3cff93aa0a98f6a4dd03733c31154b2281fa302802676c8b9cd8cb not found: ID does not exist"
Sep 16 21:53:31.667077 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.667052    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"]
Sep 16 21:53:31.671063 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.671041    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-dcbfbc-865f868fcc-lllzw"]
Sep 16 21:53:31.942836 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:31.942802    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" path="/var/lib/kubelet/pods/f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a/volumes"
Sep 16 21:53:32.115632 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:32.115586    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:32.745397 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:32.745361    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:33.534409 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:33.534374    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:34.250286 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:34.250249    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:36.519817 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:36.519774    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:37.384684 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:37.384647    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:37.485396 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:37.485360    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:37.663195 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:37.663110    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:41.869736 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:41.869702    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:42.542623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:42.542586    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:43.211368 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:43.211334    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:44.034491 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:44.034446    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:44.686371 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:44.686332    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:56.744776 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:56.744738    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"]
Sep 16 21:53:56.745302 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:56.744985    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28" podUID="203ee882-9cae-4884-8e75-3789ea89e023" containerName="maas-api" containerID="cri-o://f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25" gracePeriod=30
Sep 16 21:53:56.779144 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:56.779107    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:56.881923 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:56.881880    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:56.986765 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:56.986721    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:57.016282 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.016255    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:53:57.121239 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.121195    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:57.153304 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.153270    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-metrics-certs\") pod \"203ee882-9cae-4884-8e75-3789ea89e023\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") "
Sep 16 21:53:57.153483 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.153314    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-maas-api-tls\") pod \"203ee882-9cae-4884-8e75-3789ea89e023\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") "
Sep 16 21:53:57.153483 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.153363    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ee882-9cae-4884-8e75-3789ea89e023-openshift-service-ca\") pod \"203ee882-9cae-4884-8e75-3789ea89e023\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") "
Sep 16 21:53:57.153483 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.153430    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pq4p\" (UniqueName: \"kubernetes.io/projected/203ee882-9cae-4884-8e75-3789ea89e023-kube-api-access-5pq4p\") pod \"203ee882-9cae-4884-8e75-3789ea89e023\" (UID: \"203ee882-9cae-4884-8e75-3789ea89e023\") "
Sep 16 21:53:57.154154 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.154115    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203ee882-9cae-4884-8e75-3789ea89e023-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "203ee882-9cae-4884-8e75-3789ea89e023" (UID: "203ee882-9cae-4884-8e75-3789ea89e023"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:53:57.169742 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.169503    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203ee882-9cae-4884-8e75-3789ea89e023-kube-api-access-5pq4p" (OuterVolumeSpecName: "kube-api-access-5pq4p") pod "203ee882-9cae-4884-8e75-3789ea89e023" (UID: "203ee882-9cae-4884-8e75-3789ea89e023"). InnerVolumeSpecName "kube-api-access-5pq4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:53:57.169993 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.169950    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "203ee882-9cae-4884-8e75-3789ea89e023" (UID: "203ee882-9cae-4884-8e75-3789ea89e023"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:53:57.171654 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.171624    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "203ee882-9cae-4884-8e75-3789ea89e023" (UID: "203ee882-9cae-4884-8e75-3789ea89e023"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:53:57.254721 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.254624    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pq4p\" (UniqueName: \"kubernetes.io/projected/203ee882-9cae-4884-8e75-3789ea89e023-kube-api-access-5pq4p\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:53:57.254721 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.254657    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:53:57.254721 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.254668    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/203ee882-9cae-4884-8e75-3789ea89e023-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:53:57.254721 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.254676    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/203ee882-9cae-4884-8e75-3789ea89e023-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:53:57.356207 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.356163    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:57.521494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.521381    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:57.779441 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.779353    2567 generic.go:356] "Generic (PLEG): container finished" podID="203ee882-9cae-4884-8e75-3789ea89e023" containerID="f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25" exitCode=0
Sep 16 21:53:57.779441 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.779420    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"
Sep 16 21:53:57.779441 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.779428    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28" event={"ID":"203ee882-9cae-4884-8e75-3789ea89e023","Type":"ContainerDied","Data":"f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25"}
Sep 16 21:53:57.779974 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.779511    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28" event={"ID":"203ee882-9cae-4884-8e75-3789ea89e023","Type":"ContainerDied","Data":"8c83fd09a54f5e93218c82675a79b4e61432bd2a0aac2b17771dd841523e7252"}
Sep 16 21:53:57.779974 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.779535    2567 scope.go:117] "RemoveContainer" containerID="f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25"
Sep 16 21:53:57.789610 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.789588    2567 scope.go:117] "RemoveContainer" containerID="f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25"
Sep 16 21:53:57.789934 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:53:57.789912    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25\": container with ID starting with f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25 not found: ID does not exist" containerID="f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25"
Sep 16 21:53:57.790020 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.789942    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25"} err="failed to get container status \"f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25\": rpc error: code = NotFound desc = could not find container \"f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25\": container with ID starting with f6df8c747a76b78374483ecbf5ab07968a0ce046d8cfb9b54acbc2afcd29fa25 not found: ID does not exist"
Sep 16 21:53:57.802508 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.802481    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"]
Sep 16 21:53:57.804312 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.804291    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ad0222-7fff654f69-6hh28"]
Sep 16 21:53:57.942432 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:57.942398    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="203ee882-9cae-4884-8e75-3789ea89e023" path="/var/lib/kubelet/pods/203ee882-9cae-4884-8e75-3789ea89e023/volumes"
Sep 16 21:53:58.218551 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:58.218464    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:59.011718 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:59.011682    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:53:59.698891 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:53:59.698857    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:02.381772 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:02.381734    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:21.774700 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:21.774658    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:21.874304 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:21.874262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:39.103136 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:39.103104    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:39.887979 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:39.887941    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:45.589282 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:45.589247    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:45.684675 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:45.684640    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:45.859500 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:45.859406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:46.492396 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:46.492353    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:47.515137 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:47.515103    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:48.169401 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:48.169372    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:48.825410 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:48.825364    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:49.529681 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:49.529643    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:56.730491 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:56.730455    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"]
Sep 16 21:54:56.731004 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:56.730710    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt" podUID="0e99efab-94a3-4a94-bda5-ae76d7c22be8" containerName="maas-api" containerID="cri-o://ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069" gracePeriod=30
Sep 16 21:54:56.759811 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:56.759786    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:56.823988 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:56.823958    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:56.925456 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:56.925423    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:56.984481 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:56.984406    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:56.987821 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:56.987799    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:54:57.081629 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.081590    2567 generic.go:356] "Generic (PLEG): container finished" podID="0e99efab-94a3-4a94-bda5-ae76d7c22be8" containerID="ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069" exitCode=0
Sep 16 21:54:57.081809 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.081638    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"
Sep 16 21:54:57.081809 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.081679    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt" event={"ID":"0e99efab-94a3-4a94-bda5-ae76d7c22be8","Type":"ContainerDied","Data":"ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069"}
Sep 16 21:54:57.081809 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.081732    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt" event={"ID":"0e99efab-94a3-4a94-bda5-ae76d7c22be8","Type":"ContainerDied","Data":"3338e3f85d926b9165308bb4a0b66c9edf41ad20e99803a97428c601202b986d"}
Sep 16 21:54:57.081809 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.081755    2567 scope.go:117] "RemoveContainer" containerID="ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069"
Sep 16 21:54:57.090792 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.090773    2567 scope.go:117] "RemoveContainer" containerID="ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069"
Sep 16 21:54:57.091098 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:54:57.091078    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069\": container with ID starting with ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069 not found: ID does not exist" containerID="ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069"
Sep 16 21:54:57.091170 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.091108    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069"} err="failed to get container status \"ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069\": rpc error: code = NotFound desc = could not find container \"ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069\": container with ID starting with ca6fe71841279bf331ac6a55b298665363363cb7763143be24cc99b0e7670069 not found: ID does not exist"
Sep 16 21:54:57.093497 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.093473    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-metrics-certs\") pod \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") "
Sep 16 21:54:57.093613 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.093526    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e99efab-94a3-4a94-bda5-ae76d7c22be8-openshift-service-ca\") pod \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") "
Sep 16 21:54:57.093613 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.093550    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-maas-api-tls\") pod \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") "
Sep 16 21:54:57.093957 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.093929    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e99efab-94a3-4a94-bda5-ae76d7c22be8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e99efab-94a3-4a94-bda5-ae76d7c22be8" (UID: "0e99efab-94a3-4a94-bda5-ae76d7c22be8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:54:57.094070 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.094026    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w628x\" (UniqueName: \"kubernetes.io/projected/0e99efab-94a3-4a94-bda5-ae76d7c22be8-kube-api-access-w628x\") pod \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\" (UID: \"0e99efab-94a3-4a94-bda5-ae76d7c22be8\") "
Sep 16 21:54:57.094458 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.094427    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e99efab-94a3-4a94-bda5-ae76d7c22be8-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:54:57.095960 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.095932    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0e99efab-94a3-4a94-bda5-ae76d7c22be8" (UID: "0e99efab-94a3-4a94-bda5-ae76d7c22be8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:54:57.095960 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.095941    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e99efab-94a3-4a94-bda5-ae76d7c22be8-kube-api-access-w628x" (OuterVolumeSpecName: "kube-api-access-w628x") pod "0e99efab-94a3-4a94-bda5-ae76d7c22be8" (UID: "0e99efab-94a3-4a94-bda5-ae76d7c22be8"). InnerVolumeSpecName "kube-api-access-w628x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:54:57.096135 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.096023    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e99efab-94a3-4a94-bda5-ae76d7c22be8" (UID: "0e99efab-94a3-4a94-bda5-ae76d7c22be8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:54:57.195853 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.195815    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:54:57.195853 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.195846    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e99efab-94a3-4a94-bda5-ae76d7c22be8-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:54:57.195853 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.195856    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w628x\" (UniqueName: \"kubernetes.io/projected/0e99efab-94a3-4a94-bda5-ae76d7c22be8-kube-api-access-w628x\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:54:57.331879 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.331848    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:57.411224 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.411189    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"]
Sep 16 21:54:57.414839 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.414811    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:57.415280 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.415259    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-03eca6-869fd9f78c-8z5dt"]
Sep 16 21:54:57.942391 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:57.942361    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e99efab-94a3-4a94-bda5-ae76d7c22be8" path="/var/lib/kubelet/pods/0e99efab-94a3-4a94-bda5-ae76d7c22be8/volumes"
Sep 16 21:54:58.203970 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:58.203864    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:54:58.919846 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:54:58.919813    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:03.404457 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:03.404414    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:03.518299 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:03.518262    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:03.753062 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:03.753027    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:04.204888 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:04.204787    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:05.241136 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:05.241100    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:05.724798 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:05.724764    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:06.541640 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:06.541610    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:14.347305 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.347268    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"]
Sep 16 21:55:14.347880 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.347544    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct" podUID="5c159f43-095a-49db-af51-bb61d57258a6" containerName="maas-api" containerID="cri-o://4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2" gracePeriod=30
Sep 16 21:55:14.383182 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.383142    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:14.451727 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.451667    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:14.514212 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.514166    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:14.605081 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.605019    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:55:14.654858 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.654809    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-metrics-certs\") pod \"5c159f43-095a-49db-af51-bb61d57258a6\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") "
Sep 16 21:55:14.655036 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.654865    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c159f43-095a-49db-af51-bb61d57258a6-openshift-service-ca\") pod \"5c159f43-095a-49db-af51-bb61d57258a6\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") "
Sep 16 21:55:14.655036 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.654888    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zstct\" (UniqueName: \"kubernetes.io/projected/5c159f43-095a-49db-af51-bb61d57258a6-kube-api-access-zstct\") pod \"5c159f43-095a-49db-af51-bb61d57258a6\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") "
Sep 16 21:55:14.655113 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.655042    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-maas-api-tls\") pod \"5c159f43-095a-49db-af51-bb61d57258a6\" (UID: \"5c159f43-095a-49db-af51-bb61d57258a6\") "
Sep 16 21:55:14.655229 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.655200    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c159f43-095a-49db-af51-bb61d57258a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c159f43-095a-49db-af51-bb61d57258a6" (UID: "5c159f43-095a-49db-af51-bb61d57258a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:14.655393 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.655372    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c159f43-095a-49db-af51-bb61d57258a6-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:14.657142 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.657112    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c159f43-095a-49db-af51-bb61d57258a6" (UID: "5c159f43-095a-49db-af51-bb61d57258a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:55:14.657253 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.657137    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c159f43-095a-49db-af51-bb61d57258a6" (UID: "5c159f43-095a-49db-af51-bb61d57258a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:55:14.657253 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.657196    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c159f43-095a-49db-af51-bb61d57258a6-kube-api-access-zstct" (OuterVolumeSpecName: "kube-api-access-zstct") pod "5c159f43-095a-49db-af51-bb61d57258a6" (UID: "5c159f43-095a-49db-af51-bb61d57258a6"). InnerVolumeSpecName "kube-api-access-zstct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:14.682705 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.682668    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:14.755858 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.755822    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:14.755858 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.755855    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zstct\" (UniqueName: \"kubernetes.io/projected/5c159f43-095a-49db-af51-bb61d57258a6-kube-api-access-zstct\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:14.755858 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.755867    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c159f43-095a-49db-af51-bb61d57258a6-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:14.924972 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:14.924871    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:15.042917 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.042876    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:15.173494 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.173457    2567 generic.go:356] "Generic (PLEG): container finished" podID="5c159f43-095a-49db-af51-bb61d57258a6" containerID="4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2" exitCode=0
Sep 16 21:55:15.173663 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.173520    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"
Sep 16 21:55:15.173663 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.173535    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct" event={"ID":"5c159f43-095a-49db-af51-bb61d57258a6","Type":"ContainerDied","Data":"4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2"}
Sep 16 21:55:15.173663 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.173596    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct" event={"ID":"5c159f43-095a-49db-af51-bb61d57258a6","Type":"ContainerDied","Data":"762cfbb9b63a67cdda69561b2116f8825f7f8280b2b0fc212caadc34db1b96f8"}
Sep 16 21:55:15.173663 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.173618    2567 scope.go:117] "RemoveContainer" containerID="4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2"
Sep 16 21:55:15.183780 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.183761    2567 scope.go:117] "RemoveContainer" containerID="4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2"
Sep 16 21:55:15.184115 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:55:15.184095    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2\": container with ID starting with 4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2 not found: ID does not exist" containerID="4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2"
Sep 16 21:55:15.184185 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.184124    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2"} err="failed to get container status \"4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2\": rpc error: code = NotFound desc = could not find container \"4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2\": container with ID starting with 4cdc63c5b25068a46210b339c7f1a996732db867cdb40254d0f8b101683289a2 not found: ID does not exist"
Sep 16 21:55:15.194368 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.194343    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"]
Sep 16 21:55:15.198033 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.198008    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a7df81-7b99bf59f4-tf2ct"]
Sep 16 21:55:15.821952 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.821894    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:15.942414 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:15.942379    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c159f43-095a-49db-af51-bb61d57258a6" path="/var/lib/kubelet/pods/5c159f43-095a-49db-af51-bb61d57258a6/volumes"
Sep 16 21:55:20.959349 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:20.959315    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:21.060158 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:21.060123    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:21.279332 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:21.279305    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:21.660160 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:21.660040    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:22.471752 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:22.471716    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:23.429786 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:23.429749    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:32.001443 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.001403    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"]
Sep 16 21:55:32.001872 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.001678    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7" podUID="320c01da-fb7b-4888-8ab2-3f7df946165e" containerName="maas-api" containerID="cri-o://8cc9fc16604922e75e1959e7939d3cdc127473d62cfec00250f9f01783059c9a" gracePeriod=30
Sep 16 21:55:32.032496 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.032464    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:32.054021 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.053936    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"]
Sep 16 21:55:32.054293 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.054259    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg" podUID="e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7" containerName="payload-processing" containerID="cri-o://06eae18d20e63a661aa7855d84a0d1dcef30315a8b6fd8b276c1a60db0a4f6f9" gracePeriod=30
Sep 16 21:55:32.095297 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.095263    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:32.198518 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.198448    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:32.256403 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.256281    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:32.263251 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.263207    2567 generic.go:356] "Generic (PLEG): container finished" podID="320c01da-fb7b-4888-8ab2-3f7df946165e" containerID="8cc9fc16604922e75e1959e7939d3cdc127473d62cfec00250f9f01783059c9a" exitCode=0
Sep 16 21:55:32.263412 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.263315    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7" event={"ID":"320c01da-fb7b-4888-8ab2-3f7df946165e","Type":"ContainerDied","Data":"8cc9fc16604922e75e1959e7939d3cdc127473d62cfec00250f9f01783059c9a"}
Sep 16 21:55:32.265776 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.265748    2567 generic.go:356] "Generic (PLEG): container finished" podID="e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7" containerID="06eae18d20e63a661aa7855d84a0d1dcef30315a8b6fd8b276c1a60db0a4f6f9" exitCode=0
Sep 16 21:55:32.265926 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.265823    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg" event={"ID":"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7","Type":"ContainerDied","Data":"06eae18d20e63a661aa7855d84a0d1dcef30315a8b6fd8b276c1a60db0a4f6f9"}
Sep 16 21:55:32.303016 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.302993    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:55:32.338814 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.338779    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:55:32.417168 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.417136    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-maas-api-tls\") pod \"320c01da-fb7b-4888-8ab2-3f7df946165e\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") "
Sep 16 21:55:32.417344 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.417180    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-plugins-config-volume\") pod \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\" (UID: \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\") "
Sep 16 21:55:32.417344 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.417213    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw2j6\" (UniqueName: \"kubernetes.io/projected/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-kube-api-access-rw2j6\") pod \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\" (UID: \"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7\") "
Sep 16 21:55:32.417344 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.417242    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-metrics-certs\") pod \"320c01da-fb7b-4888-8ab2-3f7df946165e\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") "
Sep 16 21:55:32.417522 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.417356    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320c01da-fb7b-4888-8ab2-3f7df946165e-openshift-service-ca\") pod \"320c01da-fb7b-4888-8ab2-3f7df946165e\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") "
Sep 16 21:55:32.417522 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.417386    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ns2d\" (UniqueName: \"kubernetes.io/projected/320c01da-fb7b-4888-8ab2-3f7df946165e-kube-api-access-4ns2d\") pod \"320c01da-fb7b-4888-8ab2-3f7df946165e\" (UID: \"320c01da-fb7b-4888-8ab2-3f7df946165e\") "
Sep 16 21:55:32.417681 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.417649    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7" (UID: "e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:32.417795 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.417775    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320c01da-fb7b-4888-8ab2-3f7df946165e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "320c01da-fb7b-4888-8ab2-3f7df946165e" (UID: "320c01da-fb7b-4888-8ab2-3f7df946165e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 21:55:32.419503 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.419470    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "320c01da-fb7b-4888-8ab2-3f7df946165e" (UID: "320c01da-fb7b-4888-8ab2-3f7df946165e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:55:32.419503 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.419490    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "320c01da-fb7b-4888-8ab2-3f7df946165e" (UID: "320c01da-fb7b-4888-8ab2-3f7df946165e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:55:32.419788 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.419759    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-kube-api-access-rw2j6" (OuterVolumeSpecName: "kube-api-access-rw2j6") pod "e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7" (UID: "e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7"). InnerVolumeSpecName "kube-api-access-rw2j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:32.419885 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.419807    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320c01da-fb7b-4888-8ab2-3f7df946165e-kube-api-access-4ns2d" (OuterVolumeSpecName: "kube-api-access-4ns2d") pod "320c01da-fb7b-4888-8ab2-3f7df946165e" (UID: "320c01da-fb7b-4888-8ab2-3f7df946165e"). InnerVolumeSpecName "kube-api-access-4ns2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:32.518679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.518585    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/320c01da-fb7b-4888-8ab2-3f7df946165e-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:32.518679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.518620    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ns2d\" (UniqueName: \"kubernetes.io/projected/320c01da-fb7b-4888-8ab2-3f7df946165e-kube-api-access-4ns2d\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:32.518679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.518632    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:32.518679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.518640    2567 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-plugins-config-volume\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:32.518679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.518650    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw2j6\" (UniqueName: \"kubernetes.io/projected/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7-kube-api-access-rw2j6\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:32.518679 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.518658    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/320c01da-fb7b-4888-8ab2-3f7df946165e-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:32.614437 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.614393    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:32.715138 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:32.715099    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:33.272603 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.272510    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"
Sep 16 21:55:33.272603 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.272517    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7" event={"ID":"320c01da-fb7b-4888-8ab2-3f7df946165e","Type":"ContainerDied","Data":"c7b258ffb75af85346bd26525c02fbe8afe5c9fa748cdee175703718ad2dfb3d"}
Sep 16 21:55:33.272603 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.272584    2567 scope.go:117] "RemoveContainer" containerID="8cc9fc16604922e75e1959e7939d3cdc127473d62cfec00250f9f01783059c9a"
Sep 16 21:55:33.274140 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.273960    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg" event={"ID":"e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7","Type":"ContainerDied","Data":"6037e804fb8b7f7e7e002bbf324c8a2817797b37bf625a5bead2d1434ff41fd9"}
Sep 16 21:55:33.274140 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.274006    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"
Sep 16 21:55:33.285291 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.285268    2567 scope.go:117] "RemoveContainer" containerID="06eae18d20e63a661aa7855d84a0d1dcef30315a8b6fd8b276c1a60db0a4f6f9"
Sep 16 21:55:33.298880 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.298856    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"]
Sep 16 21:55:33.302729 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.302702    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-1b68db-776b8564c9-jhscg"]
Sep 16 21:55:33.312402 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.312376    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"]
Sep 16 21:55:33.316029 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.316008    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-1b68db-77b95bb555-2lxp7"]
Sep 16 21:55:33.947250 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.947211    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="320c01da-fb7b-4888-8ab2-3f7df946165e" path="/var/lib/kubelet/pods/320c01da-fb7b-4888-8ab2-3f7df946165e/volumes"
Sep 16 21:55:33.947592 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:33.947577    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7" path="/var/lib/kubelet/pods/e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7/volumes"
Sep 16 21:55:38.556319 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:38.556280    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:38.655603 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:38.655556    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:38.826096 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:38.826004    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:55:41.295251 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.295210    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d8855c556-74bwf"]
Sep 16 21:55:41.295741 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.295411    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d8855c556-74bwf" podUID="95406f41-7566-4fc1-a4c5-13ec5eddebef" containerName="manager" containerID="cri-o://b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd" gracePeriod=30
Sep 16 21:55:41.543590 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.543564    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:55:41.602550 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.602448    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-metrics-certs\") pod \"95406f41-7566-4fc1-a4c5-13ec5eddebef\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") "
Sep 16 21:55:41.602550 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.602509    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-webhook-cert\") pod \"95406f41-7566-4fc1-a4c5-13ec5eddebef\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") "
Sep 16 21:55:41.602783 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.602625    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr9kb\" (UniqueName: \"kubernetes.io/projected/95406f41-7566-4fc1-a4c5-13ec5eddebef-kube-api-access-gr9kb\") pod \"95406f41-7566-4fc1-a4c5-13ec5eddebef\" (UID: \"95406f41-7566-4fc1-a4c5-13ec5eddebef\") "
Sep 16 21:55:41.604736 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.604701    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "95406f41-7566-4fc1-a4c5-13ec5eddebef" (UID: "95406f41-7566-4fc1-a4c5-13ec5eddebef"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:55:41.605149 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.605124    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "95406f41-7566-4fc1-a4c5-13ec5eddebef" (UID: "95406f41-7566-4fc1-a4c5-13ec5eddebef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:55:41.605273 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.605156    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95406f41-7566-4fc1-a4c5-13ec5eddebef-kube-api-access-gr9kb" (OuterVolumeSpecName: "kube-api-access-gr9kb") pod "95406f41-7566-4fc1-a4c5-13ec5eddebef" (UID: "95406f41-7566-4fc1-a4c5-13ec5eddebef"). InnerVolumeSpecName "kube-api-access-gr9kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:55:41.703966 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.703874    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:41.703966 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.703958    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95406f41-7566-4fc1-a4c5-13ec5eddebef-webhook-cert\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:41.703966 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:41.703974    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9kb\" (UniqueName: \"kubernetes.io/projected/95406f41-7566-4fc1-a4c5-13ec5eddebef-kube-api-access-gr9kb\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:55:42.322731 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:42.322691    2567 generic.go:356] "Generic (PLEG): container finished" podID="95406f41-7566-4fc1-a4c5-13ec5eddebef" containerID="b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd" exitCode=0
Sep 16 21:55:42.323237 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:42.322742    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-74bwf"
Sep 16 21:55:42.323237 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:42.322777    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-74bwf" event={"ID":"95406f41-7566-4fc1-a4c5-13ec5eddebef","Type":"ContainerDied","Data":"b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd"}
Sep 16 21:55:42.323237 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:42.322833    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-74bwf" event={"ID":"95406f41-7566-4fc1-a4c5-13ec5eddebef","Type":"ContainerDied","Data":"7c12c9b5b9f0c448259275b50edd365bf6e3318f5aee45e538e10f5d3afdc35d"}
Sep 16 21:55:42.323237 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:42.322867    2567 scope.go:117] "RemoveContainer" containerID="b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd"
Sep 16 21:55:42.332182 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:42.332163    2567 scope.go:117] "RemoveContainer" containerID="b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd"
Sep 16 21:55:42.332447 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:55:42.332424    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd\": container with ID starting with b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd not found: ID does not exist" containerID="b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd"
Sep 16 21:55:42.332524 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:42.332461    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd"} err="failed to get container status \"b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd\": rpc error: code = NotFound desc = could not find container \"b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd\": container with ID starting with b6e1b736453972c9b1c55c77ad05a3b643f9179c44d0888e883ee041d95953bd not found: ID does not exist"
Sep 16 21:55:42.338658 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:42.338630    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d8855c556-74bwf"]
Sep 16 21:55:42.342202 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:42.342174    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d8855c556-74bwf"]
Sep 16 21:55:43.948373 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:55:43.948333    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95406f41-7566-4fc1-a4c5-13ec5eddebef" path="/var/lib/kubelet/pods/95406f41-7566-4fc1-a4c5-13ec5eddebef/volumes"
Sep 16 21:56:03.818973 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:03.818915    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:56:43.128630 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.128594    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d8855c556-5ljpg"]
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.128973    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203ee882-9cae-4884-8e75-3789ea89e023" containerName="maas-api"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.128987    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="203ee882-9cae-4884-8e75-3789ea89e023" containerName="maas-api"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129003    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95406f41-7566-4fc1-a4c5-13ec5eddebef" containerName="manager"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129008    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="95406f41-7566-4fc1-a4c5-13ec5eddebef" containerName="manager"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129016    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7" containerName="payload-processing"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129022    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7" containerName="payload-processing"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129030    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c159f43-095a-49db-af51-bb61d57258a6" containerName="maas-api"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129035    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c159f43-095a-49db-af51-bb61d57258a6" containerName="maas-api"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129045    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" containerName="maas-api"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129050    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" containerName="maas-api"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129062    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320c01da-fb7b-4888-8ab2-3f7df946165e" containerName="maas-api"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129067    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="320c01da-fb7b-4888-8ab2-3f7df946165e" containerName="maas-api"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129073    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e99efab-94a3-4a94-bda5-ae76d7c22be8" containerName="maas-api"
Sep 16 21:56:43.129102 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129078    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e99efab-94a3-4a94-bda5-ae76d7c22be8" containerName="maas-api"
Sep 16 21:56:43.129527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129134    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c159f43-095a-49db-af51-bb61d57258a6" containerName="maas-api"
Sep 16 21:56:43.129527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129140    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="320c01da-fb7b-4888-8ab2-3f7df946165e" containerName="maas-api"
Sep 16 21:56:43.129527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129149    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="203ee882-9cae-4884-8e75-3789ea89e023" containerName="maas-api"
Sep 16 21:56:43.129527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129154    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b6184e-b5fc-4ba9-9384-e5ba8e1adc7a" containerName="maas-api"
Sep 16 21:56:43.129527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129160    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="95406f41-7566-4fc1-a4c5-13ec5eddebef" containerName="manager"
Sep 16 21:56:43.129527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129167    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26d8bf9-fb0e-4bfe-8bd6-911e3ed4e4d7" containerName="payload-processing"
Sep 16 21:56:43.129527 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.129173    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e99efab-94a3-4a94-bda5-ae76d7c22be8" containerName="maas-api"
Sep 16 21:56:43.132181 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.132163    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.135352 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.135323    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 21:56:43.135352 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.135350    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 21:56:43.135552 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.135358    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 21:56:43.135552 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.135380    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nfgqp\""
Sep 16 21:56:43.141560 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.141536    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8855c556-5ljpg"]
Sep 16 21:56:43.278279 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.278246    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-webhook-cert\") pod \"maas-controller-6d8855c556-5ljpg\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") " pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.278451 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.278312    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-metrics-certs\") pod \"maas-controller-6d8855c556-5ljpg\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") " pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.278451 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.278386    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbr8n\" (UniqueName: \"kubernetes.io/projected/fb2e2fba-6f55-4b2c-b177-18586c19f401-kube-api-access-lbr8n\") pod \"maas-controller-6d8855c556-5ljpg\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") " pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.379754 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.379660    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbr8n\" (UniqueName: \"kubernetes.io/projected/fb2e2fba-6f55-4b2c-b177-18586c19f401-kube-api-access-lbr8n\") pod \"maas-controller-6d8855c556-5ljpg\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") " pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.379973 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.379771    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-webhook-cert\") pod \"maas-controller-6d8855c556-5ljpg\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") " pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.379973 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.379861    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-metrics-certs\") pod \"maas-controller-6d8855c556-5ljpg\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") " pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.382343 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.382321    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-metrics-certs\") pod \"maas-controller-6d8855c556-5ljpg\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") " pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.382452 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.382377    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-webhook-cert\") pod \"maas-controller-6d8855c556-5ljpg\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") " pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.387546 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.387528    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbr8n\" (UniqueName: \"kubernetes.io/projected/fb2e2fba-6f55-4b2c-b177-18586c19f401-kube-api-access-lbr8n\") pod \"maas-controller-6d8855c556-5ljpg\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") " pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.443046 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.443014    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:43.714515 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.714482    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8855c556-5ljpg"]
Sep 16 21:56:43.716731 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:56:43.716696    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2e2fba_6f55_4b2c_b177_18586c19f401.slice/crio-b94423e6aaba88615f9dfab00c6ba0f93c5cedaeeef9b9588ab0d1f7dba5e126 WatchSource:0}: Error finding container b94423e6aaba88615f9dfab00c6ba0f93c5cedaeeef9b9588ab0d1f7dba5e126: Status 404 returned error can't find the container with id b94423e6aaba88615f9dfab00c6ba0f93c5cedaeeef9b9588ab0d1f7dba5e126
Sep 16 21:56:43.718082 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.718066    2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 21:56:43.819605 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:43.819573    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:56:44.619163 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:44.619128    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-5ljpg" event={"ID":"fb2e2fba-6f55-4b2c-b177-18586c19f401","Type":"ContainerStarted","Data":"b94423e6aaba88615f9dfab00c6ba0f93c5cedaeeef9b9588ab0d1f7dba5e126"}
Sep 16 21:56:45.625599 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:45.625561    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-5ljpg" event={"ID":"fb2e2fba-6f55-4b2c-b177-18586c19f401","Type":"ContainerStarted","Data":"891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17"}
Sep 16 21:56:45.626046 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:45.625673    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:56:45.643997 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:45.643946    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d8855c556-5ljpg" podStartSLOduration=1.641894328 podStartE2EDuration="2.643927977s" totalImagesPullingTime="1.002033649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:56:45.642096473 +0000 UTC m=+2092.290508461" watchObservedRunningTime="2026-09-16 21:56:45.643927977 +0000 UTC m=+2092.292339961"
Sep 16 21:56:53.981674 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:53.981638    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:56:53.986233 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:53.986208    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 21:56:56.633403 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:56:56.633369    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:57:05.872007 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:05.871973    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:57:05.967600 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:05.967567    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d8855c556-5ljpg"]
Sep 16 21:57:05.967831 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:05.967803    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6d8855c556-5ljpg" podUID="fb2e2fba-6f55-4b2c-b177-18586c19f401" containerName="manager" containerID="cri-o://891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17" gracePeriod=30
Sep 16 21:57:06.226710 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.226677    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:57:06.277639 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.277599    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-metrics-certs\") pod \"fb2e2fba-6f55-4b2c-b177-18586c19f401\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") "
Sep 16 21:57:06.277840 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.277669    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-webhook-cert\") pod \"fb2e2fba-6f55-4b2c-b177-18586c19f401\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") "
Sep 16 21:57:06.277840 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.277765    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbr8n\" (UniqueName: \"kubernetes.io/projected/fb2e2fba-6f55-4b2c-b177-18586c19f401-kube-api-access-lbr8n\") pod \"fb2e2fba-6f55-4b2c-b177-18586c19f401\" (UID: \"fb2e2fba-6f55-4b2c-b177-18586c19f401\") "
Sep 16 21:57:06.279842 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.279809    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fb2e2fba-6f55-4b2c-b177-18586c19f401" (UID: "fb2e2fba-6f55-4b2c-b177-18586c19f401"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:57:06.279993 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.279820    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fb2e2fba-6f55-4b2c-b177-18586c19f401" (UID: "fb2e2fba-6f55-4b2c-b177-18586c19f401"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 21:57:06.280440 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.280418    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2e2fba-6f55-4b2c-b177-18586c19f401-kube-api-access-lbr8n" (OuterVolumeSpecName: "kube-api-access-lbr8n") pod "fb2e2fba-6f55-4b2c-b177-18586c19f401" (UID: "fb2e2fba-6f55-4b2c-b177-18586c19f401"). InnerVolumeSpecName "kube-api-access-lbr8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 21:57:06.379363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.379322    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:57:06.379363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.379357    2567 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fb2e2fba-6f55-4b2c-b177-18586c19f401-webhook-cert\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:57:06.379363 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.379367    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbr8n\" (UniqueName: \"kubernetes.io/projected/fb2e2fba-6f55-4b2c-b177-18586c19f401-kube-api-access-lbr8n\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 21:57:06.729506 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.729462    2567 generic.go:356] "Generic (PLEG): container finished" podID="fb2e2fba-6f55-4b2c-b177-18586c19f401" containerID="891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17" exitCode=0
Sep 16 21:57:06.729506 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.729511    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-5ljpg"
Sep 16 21:57:06.729792 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.729552    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-5ljpg" event={"ID":"fb2e2fba-6f55-4b2c-b177-18586c19f401","Type":"ContainerDied","Data":"891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17"}
Sep 16 21:57:06.729792 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.729622    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-5ljpg" event={"ID":"fb2e2fba-6f55-4b2c-b177-18586c19f401","Type":"ContainerDied","Data":"b94423e6aaba88615f9dfab00c6ba0f93c5cedaeeef9b9588ab0d1f7dba5e126"}
Sep 16 21:57:06.729792 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.729653    2567 scope.go:117] "RemoveContainer" containerID="891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17"
Sep 16 21:57:06.739311 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.739293    2567 scope.go:117] "RemoveContainer" containerID="891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17"
Sep 16 21:57:06.739579 ip-10-0-135-209 kubenswrapper[2567]: E0916 21:57:06.739563    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17\": container with ID starting with 891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17 not found: ID does not exist" containerID="891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17"
Sep 16 21:57:06.739649 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.739589    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17"} err="failed to get container status \"891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17\": rpc error: code = NotFound desc = could not find container \"891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17\": container with ID starting with 891ab260ff6d24c67266a5f219abe384528941578fa8c0b25fa1429751bfed17 not found: ID does not exist"
Sep 16 21:57:06.751030 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.750997    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6d8855c556-5ljpg"]
Sep 16 21:57:06.752826 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:06.752806    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6d8855c556-5ljpg"]
Sep 16 21:57:07.516433 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.516357    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:57:07.590154 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.590119    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6d8855c556-jqnjt"]
Sep 16 21:57:07.590535 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.590521    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb2e2fba-6f55-4b2c-b177-18586c19f401" containerName="manager"
Sep 16 21:57:07.590585 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.590537    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb2e2fba-6f55-4b2c-b177-18586c19f401" containerName="manager"
Sep 16 21:57:07.590621 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.590597    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb2e2fba-6f55-4b2c-b177-18586c19f401" containerName="manager"
Sep 16 21:57:07.595101 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.595079    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.600980 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.600958    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 21:57:07.601974 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.601932    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nfgqp\""
Sep 16 21:57:07.601974 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.601954    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 21:57:07.602098 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.601932    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 21:57:07.610966 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.610943    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8855c556-jqnjt"]
Sep 16 21:57:07.692044 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.692003    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38a361d4-a567-4c51-ba7b-120da648d5aa-webhook-cert\") pod \"maas-controller-6d8855c556-jqnjt\" (UID: \"38a361d4-a567-4c51-ba7b-120da648d5aa\") " pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.692044 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.692039    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8vz\" (UniqueName: \"kubernetes.io/projected/38a361d4-a567-4c51-ba7b-120da648d5aa-kube-api-access-dc8vz\") pod \"maas-controller-6d8855c556-jqnjt\" (UID: \"38a361d4-a567-4c51-ba7b-120da648d5aa\") " pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.692271 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.692122    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a361d4-a567-4c51-ba7b-120da648d5aa-metrics-certs\") pod \"maas-controller-6d8855c556-jqnjt\" (UID: \"38a361d4-a567-4c51-ba7b-120da648d5aa\") " pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.792623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.792538    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38a361d4-a567-4c51-ba7b-120da648d5aa-webhook-cert\") pod \"maas-controller-6d8855c556-jqnjt\" (UID: \"38a361d4-a567-4c51-ba7b-120da648d5aa\") " pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.792623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.792571    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8vz\" (UniqueName: \"kubernetes.io/projected/38a361d4-a567-4c51-ba7b-120da648d5aa-kube-api-access-dc8vz\") pod \"maas-controller-6d8855c556-jqnjt\" (UID: \"38a361d4-a567-4c51-ba7b-120da648d5aa\") " pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.792623 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.792612    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a361d4-a567-4c51-ba7b-120da648d5aa-metrics-certs\") pod \"maas-controller-6d8855c556-jqnjt\" (UID: \"38a361d4-a567-4c51-ba7b-120da648d5aa\") " pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.795069 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.795047    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38a361d4-a567-4c51-ba7b-120da648d5aa-metrics-certs\") pod \"maas-controller-6d8855c556-jqnjt\" (UID: \"38a361d4-a567-4c51-ba7b-120da648d5aa\") " pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.795195 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.795176    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/38a361d4-a567-4c51-ba7b-120da648d5aa-webhook-cert\") pod \"maas-controller-6d8855c556-jqnjt\" (UID: \"38a361d4-a567-4c51-ba7b-120da648d5aa\") " pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.800445 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.800427    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8vz\" (UniqueName: \"kubernetes.io/projected/38a361d4-a567-4c51-ba7b-120da648d5aa-kube-api-access-dc8vz\") pod \"maas-controller-6d8855c556-jqnjt\" (UID: \"38a361d4-a567-4c51-ba7b-120da648d5aa\") " pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.905234 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.905197    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:07.942365 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:07.942333    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb2e2fba-6f55-4b2c-b177-18586c19f401" path="/var/lib/kubelet/pods/fb2e2fba-6f55-4b2c-b177-18586c19f401/volumes"
Sep 16 21:57:08.174671 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:08.174644    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6d8855c556-jqnjt"]
Sep 16 21:57:08.177244 ip-10-0-135-209 kubenswrapper[2567]: W0916 21:57:08.177217    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a361d4_a567_4c51_ba7b_120da648d5aa.slice/crio-d7266abdb9b030e5230d98f056a404c85b7f9998ebbbd6e2742507c9e2f4b77a WatchSource:0}: Error finding container d7266abdb9b030e5230d98f056a404c85b7f9998ebbbd6e2742507c9e2f4b77a: Status 404 returned error can't find the container with id d7266abdb9b030e5230d98f056a404c85b7f9998ebbbd6e2742507c9e2f4b77a
Sep 16 21:57:08.741628 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:08.741583    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-jqnjt" event={"ID":"38a361d4-a567-4c51-ba7b-120da648d5aa","Type":"ContainerStarted","Data":"12bdd32014b1b63557f78f641faa13f57f08262dd532eb7f670beadc746eb7d2"}
Sep 16 21:57:08.742111 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:08.741684    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6d8855c556-jqnjt" event={"ID":"38a361d4-a567-4c51-ba7b-120da648d5aa","Type":"ContainerStarted","Data":"d7266abdb9b030e5230d98f056a404c85b7f9998ebbbd6e2742507c9e2f4b77a"}
Sep 16 21:57:08.742111 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:08.741712    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:57:08.760614 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:08.760544    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6d8855c556-jqnjt" podStartSLOduration=1.351084656 podStartE2EDuration="1.760523126s" totalImagesPullingTime="409.43847ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 21:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 21:57:08.758426528 +0000 UTC m=+2115.406838528" watchObservedRunningTime="2026-09-16 21:57:08.760523126 +0000 UTC m=+2115.408935118"
Sep 16 21:57:19.749269 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:57:19.749232    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6d8855c556-jqnjt"
Sep 16 21:59:20.634401 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:59:20.634320    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 21:59:59.616160 ip-10-0-135-209 kubenswrapper[2567]: I0916 21:59:59.616124    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:00:11.423588 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:11.423552    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:00:24.199731 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:24.199693    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:00:28.011525 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:28.011490    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:00:29.700975 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:29.700942    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:00:37.664500 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:37.664468    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:00:43.319684 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.319651    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"]
Sep 16 22:00:43.323304 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.323284    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.332855 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.332830    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"]
Sep 16 22:00:43.383329 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.383291    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65e1118-e655-49f8-94ec-5ffdc98b603f-metrics-certs\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.383516 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.383358    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvr98\" (UniqueName: \"kubernetes.io/projected/e65e1118-e655-49f8-94ec-5ffdc98b603f-kube-api-access-zvr98\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.383516 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.383438    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65e1118-e655-49f8-94ec-5ffdc98b603f-openshift-service-ca\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.383632 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.383527    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65e1118-e655-49f8-94ec-5ffdc98b603f-maas-api-tls\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.484660 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.484618    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvr98\" (UniqueName: \"kubernetes.io/projected/e65e1118-e655-49f8-94ec-5ffdc98b603f-kube-api-access-zvr98\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.484854 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.484670    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65e1118-e655-49f8-94ec-5ffdc98b603f-openshift-service-ca\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.484854 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.484710    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65e1118-e655-49f8-94ec-5ffdc98b603f-maas-api-tls\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.484854 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.484748    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65e1118-e655-49f8-94ec-5ffdc98b603f-metrics-certs\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.485359 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.485330    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65e1118-e655-49f8-94ec-5ffdc98b603f-openshift-service-ca\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.487357 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.487323    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65e1118-e655-49f8-94ec-5ffdc98b603f-maas-api-tls\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.487480 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.487323    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65e1118-e655-49f8-94ec-5ffdc98b603f-metrics-certs\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.494751 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.494727    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvr98\" (UniqueName: \"kubernetes.io/projected/e65e1118-e655-49f8-94ec-5ffdc98b603f-kube-api-access-zvr98\") pod \"maas-api-6789c486f4-4r44j\" (UID: \"e65e1118-e655-49f8-94ec-5ffdc98b603f\") " pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.635665 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.635569    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:43.911933 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:43.911880    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"]
Sep 16 22:00:43.914165 ip-10-0-135-209 kubenswrapper[2567]: W0916 22:00:43.914134    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65e1118_e655_49f8_94ec_5ffdc98b603f.slice/crio-c1e55aa167ac2e217966579c27e06d290204a14fe8a83bee44f2cd9f0375f359 WatchSource:0}: Error finding container c1e55aa167ac2e217966579c27e06d290204a14fe8a83bee44f2cd9f0375f359: Status 404 returned error can't find the container with id c1e55aa167ac2e217966579c27e06d290204a14fe8a83bee44f2cd9f0375f359
Sep 16 22:00:44.809415 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:44.809309    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j" event={"ID":"e65e1118-e655-49f8-94ec-5ffdc98b603f","Type":"ContainerStarted","Data":"ef164aa68e0fd1e555a3ded63b157cec9ed2cd112fdad91bc215aa38d81c3484"}
Sep 16 22:00:44.809415 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:44.809402    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j" event={"ID":"e65e1118-e655-49f8-94ec-5ffdc98b603f","Type":"ContainerStarted","Data":"c1e55aa167ac2e217966579c27e06d290204a14fe8a83bee44f2cd9f0375f359"}
Sep 16 22:00:44.809836 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:44.809470    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:44.826846 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:44.826794    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j" podStartSLOduration=1.242546795 podStartE2EDuration="1.826777792s" totalImagesPullingTime="584.230997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 22:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 22:00:44.826474668 +0000 UTC m=+2331.474886657" watchObservedRunningTime="2026-09-16 22:00:44.826777792 +0000 UTC m=+2331.475189779"
Sep 16 22:00:50.821421 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:50.821344    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6789c486f4-4r44j"
Sep 16 22:00:59.067673 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:00:59.067635    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:01:01.928651 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:01.928611    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"]
Sep 16 22:01:01.929091 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:01.928849    2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt" podUID="34069190-3e33-4cbe-aefd-87de44dc3a5c" containerName="maas-api" containerID="cri-o://e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53" gracePeriod=30
Sep 16 22:01:03.178949 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.178921    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 22:01:03.274646 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.274618    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-metrics-certs\") pod \"34069190-3e33-4cbe-aefd-87de44dc3a5c\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") "
Sep 16 22:01:03.274845 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.274668    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-maas-api-tls\") pod \"34069190-3e33-4cbe-aefd-87de44dc3a5c\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") "
Sep 16 22:01:03.274845 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.274701    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njg8s\" (UniqueName: \"kubernetes.io/projected/34069190-3e33-4cbe-aefd-87de44dc3a5c-kube-api-access-njg8s\") pod \"34069190-3e33-4cbe-aefd-87de44dc3a5c\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") "
Sep 16 22:01:03.274845 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.274722    2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34069190-3e33-4cbe-aefd-87de44dc3a5c-openshift-service-ca\") pod \"34069190-3e33-4cbe-aefd-87de44dc3a5c\" (UID: \"34069190-3e33-4cbe-aefd-87de44dc3a5c\") "
Sep 16 22:01:03.275174 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.275148    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34069190-3e33-4cbe-aefd-87de44dc3a5c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "34069190-3e33-4cbe-aefd-87de44dc3a5c" (UID: "34069190-3e33-4cbe-aefd-87de44dc3a5c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 22:01:03.276843 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.276817    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "34069190-3e33-4cbe-aefd-87de44dc3a5c" (UID: "34069190-3e33-4cbe-aefd-87de44dc3a5c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 22:01:03.277052 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.276839    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "34069190-3e33-4cbe-aefd-87de44dc3a5c" (UID: "34069190-3e33-4cbe-aefd-87de44dc3a5c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 22:01:03.277052 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.276981    2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34069190-3e33-4cbe-aefd-87de44dc3a5c-kube-api-access-njg8s" (OuterVolumeSpecName: "kube-api-access-njg8s") pod "34069190-3e33-4cbe-aefd-87de44dc3a5c" (UID: "34069190-3e33-4cbe-aefd-87de44dc3a5c"). InnerVolumeSpecName "kube-api-access-njg8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 22:01:03.376301 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.376265    2567 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-metrics-certs\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 22:01:03.376301 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.376294    2567 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34069190-3e33-4cbe-aefd-87de44dc3a5c-maas-api-tls\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 22:01:03.376301 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.376302    2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njg8s\" (UniqueName: \"kubernetes.io/projected/34069190-3e33-4cbe-aefd-87de44dc3a5c-kube-api-access-njg8s\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 22:01:03.376301 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.376311    2567 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34069190-3e33-4cbe-aefd-87de44dc3a5c-openshift-service-ca\") on node \"ip-10-0-135-209.ec2.internal\" DevicePath \"\""
Sep 16 22:01:03.904460 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.904426    2567 generic.go:356] "Generic (PLEG): container finished" podID="34069190-3e33-4cbe-aefd-87de44dc3a5c" containerID="e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53" exitCode=0
Sep 16 22:01:03.904661 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.904471    2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"
Sep 16 22:01:03.904661 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.904516    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt" event={"ID":"34069190-3e33-4cbe-aefd-87de44dc3a5c","Type":"ContainerDied","Data":"e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53"}
Sep 16 22:01:03.904661 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.904570    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt" event={"ID":"34069190-3e33-4cbe-aefd-87de44dc3a5c","Type":"ContainerDied","Data":"3ed7dcea472a9279aa67ad9e1bc3d4d7d07ed88c668328e89250041bc102533f"}
Sep 16 22:01:03.904661 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.904639    2567 scope.go:117] "RemoveContainer" containerID="e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53"
Sep 16 22:01:03.914143 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.914121    2567 scope.go:117] "RemoveContainer" containerID="e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53"
Sep 16 22:01:03.914420 ip-10-0-135-209 kubenswrapper[2567]: E0916 22:01:03.914403    2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53\": container with ID starting with e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53 not found: ID does not exist" containerID="e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53"
Sep 16 22:01:03.914465 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.914429    2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53"} err="failed to get container status \"e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53\": rpc error: code = NotFound desc = could not find container \"e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53\": container with ID starting with e5bdb1f60ba52ee36044a7c1744e4130227d54147a74d4ab0c943e4e681fca53 not found: ID does not exist"
Sep 16 22:01:03.925078 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.925053    2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"]
Sep 16 22:01:03.929561 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.929536    2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7d747b4f9d-jdfdt"]
Sep 16 22:01:03.943148 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:03.943112    2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34069190-3e33-4cbe-aefd-87de44dc3a5c" path="/var/lib/kubelet/pods/34069190-3e33-4cbe-aefd-87de44dc3a5c/volumes"
Sep 16 22:01:18.367322 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:18.367289    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8855c556-jqnjt_38a361d4-a567-4c51-ba7b-120da648d5aa/manager/0.log"
Sep 16 22:01:18.702532 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:18.702449    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-dkns4_5f4ab67b-5713-4d86-b4ab-b9081e9da416/copy-manifests/0.log"
Sep 16 22:01:18.708839 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:18.708811    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-dkns4_5f4ab67b-5713-4d86-b4ab-b9081e9da416/manager/0.log"
Sep 16 22:01:19.267966 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:19.267934    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6789c486f4-4r44j_e65e1118-e655-49f8-94ec-5ffdc98b603f/maas-api/0.log"
Sep 16 22:01:19.697654 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:19.697553    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-nnj72_117a9d10-99ce-473f-8b93-2b022da5f498/postgres/0.log"
Sep 16 22:01:20.557035 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:20.557005    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bdtzc_6b6404bd-dfa5-4f11-a268-f0b39315000a/manager/0.log"
Sep 16 22:01:22.008360 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:22.008324    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-454lz_3d3cee08-b372-4481-a06d-a0325f07b0e1/payload-pre-processing/0.log"
Sep 16 22:01:22.232624 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:22.232594    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f57db6868-lgg22_863ab98a-8183-4e24-aba7-8d57e12368c6/router/0.log"
Sep 16 22:01:22.577806 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:22.577772    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-z45lg_b27766ac-e448-487a-ad44-e3195975e93d/main/0.log"
Sep 16 22:01:22.792699 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:22.792667    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-2wcgf_a69d8e80-4cd1-4422-913f-cd8ddad9e621/main/0.log"
Sep 16 22:01:23.008649 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:23.008621    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fcxljp_b546a798-80ed-4923-864f-2f7c12bc7800/main/0.log"
Sep 16 22:01:23.112306 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:23.112275    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vf42p_9a8b35e1-3a52-453b-ad21-d0e03ebc7db3/main/0.log"
Sep 16 22:01:27.832385 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:27.832352    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:01:27.883928 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:27.883884    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:01:29.408282 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:29.408237    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:01:29.473607 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:29.473577    2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log" limit=1024
Sep 16 22:01:32.503548 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:32.503519    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dh4v8_38c2151a-8d75-4772-a0a2-922fb7ca7e20/global-pull-secret-syncer/0.log"
Sep 16 22:01:32.632453 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:32.632422    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f247x_7386598c-4acd-4326-8f66-f455708aaa53/konnectivity-agent/0.log"
Sep 16 22:01:32.727512 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:32.727483    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-209.ec2.internal_9292673e818dc6000ad469c880116e19/haproxy/0.log"
Sep 16 22:01:37.927515 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:37.927485    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-bdtzc_6b6404bd-dfa5-4f11-a268-f0b39315000a/manager/0.log"
Sep 16 22:01:39.838833 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:39.838784    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-5ldzz_2cfc325f-85d2-4a82-aa0e-12c2ff6beb14/cluster-monitoring-operator/0.log"
Sep 16 22:01:40.031190 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:40.031161    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cz9qx_fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5/node-exporter/0.log"
Sep 16 22:01:40.066326 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:40.066284    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cz9qx_fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5/kube-rbac-proxy/0.log"
Sep 16 22:01:40.106217 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:40.106190    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cz9qx_fcd949d4-ff94-4417-b3d8-cc60e1cf5ea5/init-textfile/0.log"
Sep 16 22:01:40.636519 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:40.636434    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-7jmwp_0ce2ec2d-3586-41bc-8db4-a15671fe8c4d/prometheus-operator-admission-webhook/0.log"
Sep 16 22:01:42.160444 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.160408    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-ndwmx_8f40f29c-8841-4a0b-8b1a-343e91436f77/networking-console-plugin/0.log"
Sep 16 22:01:42.750478 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.750448    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 22:01:42.754320 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.754297    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/2.log"
Sep 16 22:01:42.977139 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.977102    2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"]
Sep 16 22:01:42.977493 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.977481    2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34069190-3e33-4cbe-aefd-87de44dc3a5c" containerName="maas-api"
Sep 16 22:01:42.977546 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.977494    2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="34069190-3e33-4cbe-aefd-87de44dc3a5c" containerName="maas-api"
Sep 16 22:01:42.977586 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.977561    2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="34069190-3e33-4cbe-aefd-87de44dc3a5c" containerName="maas-api"
Sep 16 22:01:42.980953 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.980932    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:42.983463 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.983439    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfn8w\"/\"openshift-service-ca.crt\""
Sep 16 22:01:42.983578 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.983436    2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pfn8w\"/\"default-dockercfg-zfj9l\""
Sep 16 22:01:42.984452 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.984434    2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pfn8w\"/\"kube-root-ca.crt\""
Sep 16 22:01:42.990544 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:42.990520    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"]
Sep 16 22:01:43.137883 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.137784    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-proc\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.137883 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.137863    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-podres\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.138136 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.137892    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-sys\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.138136 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.138000    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-lib-modules\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.138136 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.138075    2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprnp\" (UniqueName: \"kubernetes.io/projected/c98a7a26-50d0-4290-8136-c1fc7567e924-kube-api-access-fprnp\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.238962 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.238897    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-proc\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.239421 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.238993    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-podres\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.239421 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.239020    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-sys\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.239421 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.239032    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-proc\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.239421 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.239100    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-lib-modules\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.239421 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.239117    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-sys\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.239421 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.239159    2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprnp\" (UniqueName: \"kubernetes.io/projected/c98a7a26-50d0-4290-8136-c1fc7567e924-kube-api-access-fprnp\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.239421 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.239166    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-podres\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.239421 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.239225    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c98a7a26-50d0-4290-8136-c1fc7567e924-lib-modules\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.247042 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.247022    2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprnp\" (UniqueName: \"kubernetes.io/projected/c98a7a26-50d0-4290-8136-c1fc7567e924-kube-api-access-fprnp\") pod \"perf-node-gather-daemonset-fhnr2\" (UID: \"c98a7a26-50d0-4290-8136-c1fc7567e924\") " pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.292245 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.292212    2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:43.324056 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.324023    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5ddcb64c95-dvrhh_13b099cb-ff08-465b-8be2-00b74b0977b1/console/0.log"
Sep 16 22:01:43.361329 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.361261    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-zjhk6_743eaf68-5cad-4048-a581-0c350e2969ce/download-server/0.log"
Sep 16 22:01:43.564548 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:43.564511    2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"]
Sep 16 22:01:43.567033 ip-10-0-135-209 kubenswrapper[2567]: W0916 22:01:43.567003    2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc98a7a26_50d0_4290_8136_c1fc7567e924.slice/crio-4aa7885c5ca5b65c338810035cf642fcd202e41630b2780464658baa8e21ed7d WatchSource:0}: Error finding container 4aa7885c5ca5b65c338810035cf642fcd202e41630b2780464658baa8e21ed7d: Status 404 returned error can't find the container with id 4aa7885c5ca5b65c338810035cf642fcd202e41630b2780464658baa8e21ed7d
Sep 16 22:01:44.030267 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:44.030241    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-skqzn_9166aeb5-51d3-4a3e-8dbb-c6ff8b46ac1a/volume-data-source-validator/0.log"
Sep 16 22:01:44.096200 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:44.096166    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2" event={"ID":"c98a7a26-50d0-4290-8136-c1fc7567e924","Type":"ContainerStarted","Data":"ad509d0f0079346ab5ae31d79e6e88e8bfd83a392e827f4d2fa424d6d791b289"}
Sep 16 22:01:44.096378 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:44.096215    2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2" event={"ID":"c98a7a26-50d0-4290-8136-c1fc7567e924","Type":"ContainerStarted","Data":"4aa7885c5ca5b65c338810035cf642fcd202e41630b2780464658baa8e21ed7d"}
Sep 16 22:01:44.096378 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:44.096246    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:44.112341 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:44.112292    2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2" podStartSLOduration=2.112275121 podStartE2EDuration="2.112275121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 22:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 22:01:44.110396367 +0000 UTC m=+2390.758808355" watchObservedRunningTime="2026-09-16 22:01:44.112275121 +0000 UTC m=+2390.760687153"
Sep 16 22:01:45.154174 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:45.154145    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45xsc_1e8049ff-a69e-4607-b24f-b98b40f53fc2/dns/0.log"
Sep 16 22:01:45.176414 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:45.176382    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45xsc_1e8049ff-a69e-4607-b24f-b98b40f53fc2/kube-rbac-proxy/0.log"
Sep 16 22:01:45.338063 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:45.338032    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xd599_5fb0449b-4fbe-446e-8aa0-9d2c653d440d/dns-node-resolver/0.log"
Sep 16 22:01:45.890931 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:45.889534    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kfcgt_b45b7d4a-6f16-42a0-b70d-7e63ca3efdd5/node-ca/0.log"
Sep 16 22:01:47.732262 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:47.732224    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-454lz_3d3cee08-b372-4481-a06d-a0325f07b0e1/payload-pre-processing/0.log"
Sep 16 22:01:47.811992 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:47.811959    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f57db6868-lgg22_863ab98a-8183-4e24-aba7-8d57e12368c6/router/0.log"
Sep 16 22:01:48.726482 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:48.726445    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhhbw_5c2c6551-1309-467a-8886-a3c278d89c03/serve-healthcheck-canary/0.log"
Sep 16 22:01:49.271631 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:49.271598    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-s2szh_9f518039-df50-4fbb-9d28-f6e554a1a784/insights-operator/0.log"
Sep 16 22:01:49.272147 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:49.272073    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-s2szh_9f518039-df50-4fbb-9d28-f6e554a1a784/insights-operator/1.log"
Sep 16 22:01:49.420632 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:49.420595    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdnb5_9093c52e-a8f7-4d7b-858f-34d04fea5ebd/kube-rbac-proxy/0.log"
Sep 16 22:01:49.439917 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:49.439877    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdnb5_9093c52e-a8f7-4d7b-858f-34d04fea5ebd/exporter/0.log"
Sep 16 22:01:49.459565 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:49.459540    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tdnb5_9093c52e-a8f7-4d7b-858f-34d04fea5ebd/extractor/0.log"
Sep 16 22:01:50.111599 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:50.111573    2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pfn8w/perf-node-gather-daemonset-fhnr2"
Sep 16 22:01:52.179214 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:52.179135    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6d8855c556-jqnjt_38a361d4-a567-4c51-ba7b-120da648d5aa/manager/0.log"
Sep 16 22:01:52.353177 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:52.353152    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-dkns4_5f4ab67b-5713-4d86-b4ab-b9081e9da416/manager/0.log"
Sep 16 22:01:52.375500 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:52.375476    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df9d657c6-dkns4_5f4ab67b-5713-4d86-b4ab-b9081e9da416/copy-manifests/0.log"
Sep 16 22:01:53.814205 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:53.814161    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5dfdd9df77-8nh9g_1ca96afd-9e20-4557-8b0a-a6f1e2d14569/manager/0.log"
Sep 16 22:01:54.019392 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:54.019361    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 22:01:54.023240 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:54.023215    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-8p9nq_f2378ae2-b1f3-41b3-a99c-1aa20b008f68/console-operator/1.log"
Sep 16 22:01:54.290796 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:54.290748    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-454lz_3d3cee08-b372-4481-a06d-a0325f07b0e1/payload-pre-processing/0.log"
Sep 16 22:01:54.339460 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:54.339430    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f57db6868-lgg22_863ab98a-8183-4e24-aba7-8d57e12368c6/router/0.log"
Sep 16 22:01:56.664648 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:56.664619    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-454lz_3d3cee08-b372-4481-a06d-a0325f07b0e1/payload-pre-processing/0.log"
Sep 16 22:01:56.714319 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:56.714290    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5f57db6868-lgg22_863ab98a-8183-4e24-aba7-8d57e12368c6/router/0.log"
Sep 16 22:01:58.955403 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:58.955367    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vv6hz_a64f28e2-ac14-4d5a-a440-f602bca5222e/migrator/0.log"
Sep 16 22:01:58.981853 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:58.981822    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-vv6hz_a64f28e2-ac14-4d5a-a440-f602bca5222e/graceful-termination/0.log"
Sep 16 22:01:59.332567 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:59.332536    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-g9pqs_2908a138-2d0e-4eb8-8b35-742e861fb491/kube-storage-version-migrator-operator/1.log"
Sep 16 22:01:59.333431 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:01:59.333415    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-g9pqs_2908a138-2d0e-4eb8-8b35-742e861fb491/kube-storage-version-migrator-operator/0.log"
Sep 16 22:02:00.440654 ip-10-0-135-209 kubenswrapper[2567]: I0916 22:02:00.440613    2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hgq4m_77e43015-e830-4e21-afed-16ddd27263e9/kube-multus-additional-cni-plugins/0.log"