Sep 10 13:18:14.882486 ip-10-0-138-19 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 13:18:15.216252 ip-10-0-138-19 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 13:18:15.216252 ip-10-0-138-19 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 13:18:15.216252 ip-10-0-138-19 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 13:18:15.216252 ip-10-0-138-19 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 13:18:15.216252 ip-10-0-138-19 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 13:18:15.217015 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.216920    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 13:18:15.219748 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219731    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:18:15.219748 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219748    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219752    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219755    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219759    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219763    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219766    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219769    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219772    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219775    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219778    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219781    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219784    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219787    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219789    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219792    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219795    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219797    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219800    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219804    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:18:15.219809 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219808    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219811    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219814    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219817    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219820    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219823    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219826    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219830    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219833    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219836    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219838    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219841    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219844    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219846    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219849    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219852    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219854    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219857    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219859    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219862    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:18:15.220282 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219864    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219867    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219871    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219874    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219877    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219879    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219882    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219885    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219888    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219892    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219898    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219901    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219904    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219907    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219911    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219914    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219917    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219919    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219922    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:18:15.220796 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219925    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219928    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219930    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219933    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219936    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219940    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219943    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219945    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219948    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219951    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219953    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219955    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219958    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219960    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219963    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219966    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219968    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219970    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219973    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219975    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:18:15.221281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219979    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219981    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219984    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219986    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219989    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219992    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219995    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.219998    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220001    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220003    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220006    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220009    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220011    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220014    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220016    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220019    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220021    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220024    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220027    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:18:15.221761 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220029    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220759    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220765    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220769    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220771    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220774    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220777    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220779    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220782    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220785    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220787    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220789    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220792    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220795    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220798    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220802    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220805    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220808    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220811    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:18:15.222234 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220813    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220816    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220819    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220821    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220824    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220826    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220829    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220831    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220834    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220837    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220839    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220842    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220844    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220847    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220850    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220853    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220855    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220858    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220862    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:18:15.222713 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220866    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220869    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220872    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220875    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220877    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220880    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220883    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220885    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220888    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220892    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220894    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220897    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220900    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220903    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220905    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220908    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220911    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220913    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220916    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220918    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:18:15.223240 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220921    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220923    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220926    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220928    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220931    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220933    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220936    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220939    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220941    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220943    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220946    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220949    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220952    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220956    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220959    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220962    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220965    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220968    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220971    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:18:15.223734 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220974    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220977    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220979    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220983    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220985    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220990    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220993    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220995    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.220998    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221000    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221003    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221005    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221008    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221010    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221013    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221016    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221019    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221022    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221024    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221027    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:18:15.224210 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221030    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221034    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221037    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221452    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221457    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221462    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221466    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221469    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221472    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221474    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221477    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221480    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221482    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221485    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221488    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221491    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221494    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221498    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221501    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:18:15.224706 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221505    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221510    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221513    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221516    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221519    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221522    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221525    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221527    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221530    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221532    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221535    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221537    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221540    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221543    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221545    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221548    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221551    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221554    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221556    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221559    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:18:15.225173 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221561    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221564    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221568    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221571    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221574    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221577    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221580    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221583    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221586    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221589    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221594    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221597    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221599    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221603    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221606    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221608    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221611    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221613    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221616    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:18:15.225672 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221618    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221620    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221623    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221626    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221628    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221630    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221633    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221635    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221638    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221641    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221643    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221646    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221649    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221651    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221654    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221656    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221658    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221661    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221663    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221666    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:18:15.226141 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221668    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221670    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221673    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221677    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221680    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221683    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221686    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221689    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221691    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221694    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221696    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221699    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221702    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221704    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221707    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221709    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221713    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221715    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221718    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221721    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:18:15.226648 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221723    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221726    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221728    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.221731    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221796    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221840    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221848    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221853    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221858    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221861    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221865    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221870    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221873    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221876    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221879    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221884    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221888    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221891    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221895    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221898    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221901    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221904    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221908    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221912    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 13:18:15.227178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221915    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221918    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221921    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221925    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221928    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221931    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221934    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221937    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221940    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221943    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221946    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221949    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221953    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221956    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221959    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221962    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221965    2578 flags.go:64] FLAG: --enable-server="true"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221967    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221972    2578 flags.go:64] FLAG: --event-burst="100"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221975    2578 flags.go:64] FLAG: --event-qps="50"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221978    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221981    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221984    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221989    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 13:18:15.227809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221992    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221995    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.221999    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222010    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222014    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222017    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222020    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222023    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222026    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222029    2578 flags.go:64] FLAG: --feature-gates=""
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222033    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222036    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222039    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222042    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222045    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222048    2578 flags.go:64] FLAG: --help="false"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222051    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222054    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222057    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222060    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222063    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222066    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222069    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 13:18:15.228484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222071    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222074    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222077    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222080    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222083    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222086    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222089    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222092    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222096    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222098    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222101    2578 flags.go:64] FLAG: --lock-file=""
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222105    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222108    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222111    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222116    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222119    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222122    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222125    2578 flags.go:64] FLAG: --logging-format="text"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222128    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222131    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222133    2578 flags.go:64] FLAG: --manifest-url=""
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222136    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222140    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222143    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222147    2578 flags.go:64] FLAG: --max-pods="110"
Sep 10 13:18:15.229557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222150    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222153    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222155    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222158    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222161    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222164    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222167    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222175    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222178    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222180    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222183    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222186    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222205    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222209    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222212    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222215    2578 flags.go:64] FLAG: --port="10250"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222219    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222222    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a4579b5497782df3"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222225    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222232    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222235    2578 flags.go:64] FLAG: --register-node="true"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222238    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222242    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222245    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 10 13:18:15.230653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222248    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222251    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222255    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222258    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222261    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222264    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222267    2578 flags.go:64] FLAG: --runonce="false"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222270    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222273    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222276    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222279    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222282    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222285    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222288    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222291    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222294    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222296    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222299    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222302    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222305    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222308    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222310    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222316    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222318    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222323    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 13:18:15.231803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222328    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222332    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222335    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222338    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222341    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222344    2578 flags.go:64] FLAG: --v="2"
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222348    2578 flags.go:64] FLAG: --version="false"
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222352    2578 flags.go:64] FLAG: --vmodule=""
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222356    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.222359    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222487    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222491    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222494    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222497    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222500    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222503    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222505    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222508    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222511    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222513    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222516    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222518    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:18:15.232637 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222521    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222525    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222529    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222532    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222535    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222538    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222541    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222543    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222546    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222549    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222552    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222556    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222559    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222561    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222564    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222567    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222570    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222572    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222575    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222577    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:18:15.233545 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222580    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222583    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222585    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222587    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222590    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222593    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222595    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222598    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222600    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222602    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222605    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222608    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222611    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222613    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222616    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222619    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222621    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222624    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222627    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222629    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:18:15.234274 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222632    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222635    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222639    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222644    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222647    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222650    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222653    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222656    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222658    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222661    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222664    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222666    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222669    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222672    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222674    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222677    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222680    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222682    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222685    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:18:15.234836 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222687    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222690    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222692    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222695    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222697    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222700    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222702    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222705    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222708    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222712    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222715    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222718    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222721    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222725    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222728    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222731    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222735    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222738    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222740    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:18:15.235330 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222743    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222746    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222750    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222752    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222755    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222758    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222760    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222763    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.222766    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.223216    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.230697    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.230716    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.230743    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.230752    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230837    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:18:15.235972 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230845    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230850    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230856    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230860    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230865    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230869    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230874    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230878    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230884    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230890    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230895    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230900    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230904    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230909    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230913    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230918    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230922    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230926    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230932    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:18:15.236481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230939    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230944    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230948    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230952    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230957    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230961    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230965    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230969    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230973    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230978    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230982    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230986    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230990    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.230995    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231000    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231003    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231007    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231012    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231016    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:18:15.236947 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231020    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231025    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231029    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231034    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231038    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231042    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231046    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231050    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231054    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231059    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231063    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231067    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231071    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231075    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231080    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231085    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231089    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231093    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231097    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231101    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:18:15.237472 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231104    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231108    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231113    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231117    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231121    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231125    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231129    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231133    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231137    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231141    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231145    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231149    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231153    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231158    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231163    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231167    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231171    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231175    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231179    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231183    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:18:15.237986 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231187    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231209    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231214    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231217    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231221    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231224    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231228    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231235    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231240    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231246    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231250    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231253    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231257    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231261    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231265    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231268    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231272    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231276    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231280    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:18:15.238868 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231283    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.231291    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231476    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231484    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231489    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231493    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231497    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231501    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231506    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231510    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231514    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231518    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231522    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231526    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231530    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 13:18:15.239655 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231534    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231540    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231545    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231550    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231554    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231559    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231564    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231568    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231572    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231576    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231581    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231585    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231588    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231593    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231597    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231601    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231605    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231609    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231613    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 13:18:15.240042 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231617    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231621    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231625    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231629    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231633    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231637    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231641    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231646    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231653    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231659    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231663    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231668    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231672    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231676    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231680    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231685    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231689    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231693    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231697    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 13:18:15.240555 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231701    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231709    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231714    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231719    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231723    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231728    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231733    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231737    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231741    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231745    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231749    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231753    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231757    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231761    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231765    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231769    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231773    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231777    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231781    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231785    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 13:18:15.246670 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231789    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231794    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231798    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231802    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231805    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231809    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231813    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231817    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231821    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231824    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231828    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231832    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231836    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231840    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231845    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231849    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231853    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231860    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231864    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231869    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 13:18:15.247157 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231873    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231877    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231881    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231885    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231889    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231893    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231898    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:15.231902    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.231910    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.232058    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.235550    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.236347    2578 server.go:1019] "Starting client certificate rotation"
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.236451    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:18:15.247833 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.236488    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:18:15.248169 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.238693    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 13:18:15.248169 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.244618    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 13:18:15.252572 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.252554    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 10 13:18:15.256615 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.256598    2578 log.go:25] "Validated CRI v1 image API"
Sep 10 13:18:15.257965 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.257949    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 13:18:15.259009 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.258989    2578 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 10 13:18:15.261234 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.261210    2578 fs.go:135] Filesystem UUIDs: map[32b79910-7e64-4cc6-9b49-ce40f1bf7e68:/dev/nvme0n1p3 476168f3-cbaa-47f1-8ca9-4dcb84b0d850:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 10 13:18:15.261284 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.261235    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 13:18:15.266826 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.266720    2578 manager.go:216] Machine: {Timestamp:2026-09-10 13:18:15.265651499 +0000 UTC m=+0.282483462 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098726 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2811c99c16f9ed1edfc0a4433b2278 SystemUUID:ec2811c9-9c16-f9ed-1edf-c0a4433b2278 BootID:13dbe7a3-8c67-410d-ae18-869777e3811f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a7:d5:83:03:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a7:d5:83:03:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:3b:c0:f4:d1:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 13:18:15.266826 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.266822    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 13:18:15.266932 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.266901    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 13:18:15.268977 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.268958    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 13:18:15.269106 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.268979    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-19.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 13:18:15.269555 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.269546    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 13:18:15.269586 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.269558    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 13:18:15.269586 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.269566    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 13:18:15.270572 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.270561    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 13:18:15.271258 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.271249    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 13:18:15.274162 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.274152    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 13:18:15.274288 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.274279    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 13:18:15.274320 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.274312    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 13:18:15.274352 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.274328    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 13:18:15.274352 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.274343    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 13:18:15.274352 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.274352    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 10 13:18:15.274431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.274361    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 13:18:15.274703 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.274693    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 13:18:15.274738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.274711    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 13:18:15.276778 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.276759    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 13:18:15.278654 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.278640    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 13:18:15.278706 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.278670    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 13:18:15.279235 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279223    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 13:18:15.279281 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279239    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 13:18:15.279281 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279254    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 13:18:15.279281 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279260    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 13:18:15.279281 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279268    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 13:18:15.279281 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279274    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 13:18:15.279424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279287    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 13:18:15.279424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279293    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 13:18:15.279424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279300    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 13:18:15.279424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279306    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 13:18:15.279424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279319    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 13:18:15.279424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279325    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 13:18:15.279424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279351    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 13:18:15.279424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.279357    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 13:18:15.283181 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.283164    2578 server.go:1268] "Started kubelet"
Sep 10 13:18:15.283279 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.283251    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 13:18:15.283405 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.283323    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 13:18:15.283459 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.283429    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 13:18:15.287245 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.287221    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:15.287339 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.287270    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 13:18:15.287406 ip-10-0-138-19 systemd[1]: Started Kubernetes Kubelet.
Sep 10 13:18:15.287978 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.287959    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 13:18:15.289097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.289074    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 13:18:15.289369 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.289341    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:18:15.289462 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.289345    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:18:15.291133 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.290309    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f820751484b0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.28313976 +0000 UTC m=+0.299971723,LastTimestamp:2026-09-10 13:18:15.28313976 +0000 UTC m=+0.299971723,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.294733 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.294717    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 13:18:15.294820 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.294731    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 13:18:15.294820 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.294801    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:18:15.295394 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295372    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 13:18:15.295394 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295379    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 13:18:15.295524 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295406    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 13:18:15.295524 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295476    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 13:18:15.295524 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295484    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 13:18:15.295651 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295566    2578 factory.go:55] Registering systemd factory
Sep 10 13:18:15.295651 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295582    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 10 13:18:15.295802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295786    2578 factory.go:153] Registering CRI-O factory
Sep 10 13:18:15.295861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295805    2578 factory.go:223] Registration of the crio container factory successfully
Sep 10 13:18:15.295861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295852    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 13:18:15.295955 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295872    2578 factory.go:103] Registering Raw factory
Sep 10 13:18:15.295955 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.295899    2578 manager.go:1196] Started watching for new ooms in manager
Sep 10 13:18:15.296301 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.296289    2578 manager.go:349] Starting recovery of all containers
Sep 10 13:18:15.298592 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.298560    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 13:18:15.298687 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.298644    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:18:15.305572 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.305555    2578 manager.go:354] Recovery completed
Sep 10 13:18:15.310614 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.310599    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:15.312428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.312395    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:15.312501 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.312448    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:15.312501 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.312466    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:15.314425 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.314410    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 13:18:15.314425 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.314421    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 13:18:15.314527 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.314436    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 13:18:15.314527 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.314486    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:15.315538 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.314917    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d36e6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312428652 +0000 UTC m=+0.329260625,LastTimestamp:2026-09-10 13:18:15.312428652 +0000 UTC m=+0.329260625,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.315645 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.315625    2578 policy_none.go:49] "None policy: Start"
Sep 10 13:18:15.315697 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.315670    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 13:18:15.315697 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.315685    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 13:18:15.317443 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.317325    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:15.317581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.317566    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:15.317581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.317575    2578 policy_none.go:47] "Start"
Sep 10 13:18:15.317731 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.317586    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:15.326252 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.326161    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d3dba3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312456611 +0000 UTC m=+0.329288576,LastTimestamp:2026-09-10 13:18:15.312456611 +0000 UTC m=+0.329288576,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.328351 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.328292    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d4169f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312471711 +0000 UTC m=+0.329303674,LastTimestamp:2026-09-10 13:18:15.312471711 +0000 UTC m=+0.329303674,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.334397 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.334338    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d36e6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d36e6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312428652 +0000 UTC m=+0.329260625,LastTimestamp:2026-09-10 13:18:15.317547159 +0000 UTC m=+0.334379124,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.336866 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.336807    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d3dba3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d3dba3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312456611 +0000 UTC m=+0.329288576,LastTimestamp:2026-09-10 13:18:15.317577722 +0000 UTC m=+0.334409698,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.342850 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.342837    2578 manager.go:337] "Starting Device Plugin manager"
Sep 10 13:18:15.342938 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.342877    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 13:18:15.342938 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.342887    2578 server.go:87] "Starting device plugin registration server"
Sep 10 13:18:15.343172 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.343162    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 13:18:15.343240 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.343176    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 13:18:15.343407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.343395    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 13:18:15.343506 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.343492    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 13:18:15.343506 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.343504    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 13:18:15.343998 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.343968    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 13:18:15.344096 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.344027    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:18:15.344413 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.344294    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d4169f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d4169f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312471711 +0000 UTC m=+0.329303674,LastTimestamp:2026-09-10 13:18:15.317592514 +0000 UTC m=+0.334424480,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.360256 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.357521    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82079674ac7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.355673287 +0000 UTC m=+0.372505240,LastTimestamp:2026-09-10 13:18:15.355673287 +0000 UTC m=+0.372505240,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.443541 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.443513    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:15.443541 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.443512    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 13:18:15.444809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.444783    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 13:18:15.444910 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.444851    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 13:18:15.444910 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.444878    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 13:18:15.445021 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.444955    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 13:18:15.445420 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.445397    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:15.445534 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.445434    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:15.445534 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.445449    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:15.445534 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.445482    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.454860 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.454773    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d36e6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d36e6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312428652 +0000 UTC m=+0.329260625,LastTimestamp:2026-09-10 13:18:15.44541981 +0000 UTC m=+0.462251774,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.454975 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.454876    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:18:15.457177 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.457158    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.457457 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.457401    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d3dba3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d3dba3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312456611 +0000 UTC m=+0.329288576,LastTimestamp:2026-09-10 13:18:15.445440682 +0000 UTC m=+0.462272652,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.463325 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.463261    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d4169f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d4169f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312471711 +0000 UTC m=+0.329303674,LastTimestamp:2026-09-10 13:18:15.445454526 +0000 UTC m=+0.462286491,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.503130 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.503043    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Sep 10 13:18:15.545052 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.545025    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"]
Sep 10 13:18:15.545155 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.545122    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:15.546619 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.546598    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:15.546727 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.546635    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:15.546727 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.546651    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:15.547905 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.547892    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:15.548025 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.548011    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.548062 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.548041    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:15.548553 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.548538    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:15.548636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.548568    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:15.548636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.548570    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:15.548636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.548582    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:15.548636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.548592    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:15.548636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.548604    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:15.549646 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.549630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.549735 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.549657    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:15.550287 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.550271    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:15.550373 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.550295    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:15.550373 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.550306    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:15.557482 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.557412    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d36e6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d36e6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312428652 +0000 UTC m=+0.329260625,LastTimestamp:2026-09-10 13:18:15.546618208 +0000 UTC m=+0.563450178,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.567035 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.566968    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d3dba3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d3dba3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312456611 +0000 UTC m=+0.329288576,LastTimestamp:2026-09-10 13:18:15.546642505 +0000 UTC m=+0.563474473,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.571803 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.571784    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.575608 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.575543    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d4169f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d4169f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312471711 +0000 UTC m=+0.329303674,LastTimestamp:2026-09-10 13:18:15.546656522 +0000 UTC m=+0.563488487,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.576263 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.576247    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.584944 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.584884    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d36e6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d36e6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312428652 +0000 UTC m=+0.329260625,LastTimestamp:2026-09-10 13:18:15.548552188 +0000 UTC m=+0.565384161,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.597097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.597074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102348c45e238ec4cfbabd8b1ecec91f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-19.ec2.internal\" (UID: \"102348c45e238ec4cfbabd8b1ecec91f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.597097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.597096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24e19692c959ce3358ef1453cec7edef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal\" (UID: \"24e19692c959ce3358ef1453cec7edef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.597254 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.597114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24e19692c959ce3358ef1453cec7edef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal\" (UID: \"24e19692c959ce3358ef1453cec7edef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.598050 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.597980    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d3dba3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d3dba3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312456611 +0000 UTC m=+0.329288576,LastTimestamp:2026-09-10 13:18:15.548574525 +0000 UTC m=+0.565406493,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.606912 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.606856    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d36e6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d36e6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312428652 +0000 UTC m=+0.329260625,LastTimestamp:2026-09-10 13:18:15.548583334 +0000 UTC m=+0.565415297,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.618708 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.618645    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d4169f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d4169f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312471711 +0000 UTC m=+0.329303674,LastTimestamp:2026-09-10 13:18:15.548588225 +0000 UTC m=+0.565420189,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.627089 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.627029    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d3dba3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d3dba3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312456611 +0000 UTC m=+0.329288576,LastTimestamp:2026-09-10 13:18:15.548596482 +0000 UTC m=+0.565428443,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.636390 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.636329    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d4169f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d4169f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312471711 +0000 UTC m=+0.329303674,LastTimestamp:2026-09-10 13:18:15.548607401 +0000 UTC m=+0.565439363,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.647368 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.647313    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d36e6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d36e6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312428652 +0000 UTC m=+0.329260625,LastTimestamp:2026-09-10 13:18:15.550284582 +0000 UTC m=+0.567116550,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.655717 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.655665    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d3dba3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d3dba3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312456611 +0000 UTC m=+0.329288576,LastTimestamp:2026-09-10 13:18:15.550300048 +0000 UTC m=+0.567132011,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.658245 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.658230    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:15.659062 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.659049    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:15.659113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.659075    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:15.659113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.659085    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:15.659113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.659111    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.664167 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.664108    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d4169f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d4169f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312471711 +0000 UTC m=+0.329303674,LastTimestamp:2026-09-10 13:18:15.550311699 +0000 UTC m=+0.567143668,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.667850 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.667829    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.667938 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.667818    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d36e6c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d36e6c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312428652 +0000 UTC m=+0.329260625,LastTimestamp:2026-09-10 13:18:15.659062482 +0000 UTC m=+0.675894447,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.672579 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.672522    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-19.ec2.internal.18d3f82076d3dba3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-19.ec2.internal.18d3f82076d3dba3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-19.ec2.internal,UID:ip-10-0-138-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-19.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:15.312456611 +0000 UTC m=+0.329288576,LastTimestamp:2026-09-10 13:18:15.65907931 +0000 UTC m=+0.675911272,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:15.697804 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.697773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102348c45e238ec4cfbabd8b1ecec91f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-19.ec2.internal\" (UID: \"102348c45e238ec4cfbabd8b1ecec91f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.697887 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.697810    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24e19692c959ce3358ef1453cec7edef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal\" (UID: \"24e19692c959ce3358ef1453cec7edef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.697887 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.697825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24e19692c959ce3358ef1453cec7edef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal\" (UID: \"24e19692c959ce3358ef1453cec7edef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.697951 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.697880    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102348c45e238ec4cfbabd8b1ecec91f-config\") pod \"kube-apiserver-proxy-ip-10-0-138-19.ec2.internal\" (UID: \"102348c45e238ec4cfbabd8b1ecec91f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.697951 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.697938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24e19692c959ce3358ef1453cec7edef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal\" (UID: \"24e19692c959ce3358ef1453cec7edef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.697951 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.697942    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/24e19692c959ce3358ef1453cec7edef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal\" (UID: \"24e19692c959ce3358ef1453cec7edef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.873872 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.873791    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.879467 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:15.879442    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"
Sep 10 13:18:15.905576 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:15.905547    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms"
Sep 10 13:18:16.068286 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.068244    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:16.069394 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.069374    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:16.069524 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.069411    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:16.069524 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.069426    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:16.069524 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.069461    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:16.090303 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:16.090271    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:16.185515 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:16.185448    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:18:16.219131 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:16.219098    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:18:16.301813 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.301785    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:16.369122 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.369090    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:16.370260 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.370243    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:16.370378 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.370277    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:16.370378 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.370287    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:16.669086 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:16.669048    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102348c45e238ec4cfbabd8b1ecec91f.slice/crio-2d84b786dce484bcab5a60855fe4b78122196e0453ee956046314b30d57bddde WatchSource:0}: Error finding container 2d84b786dce484bcab5a60855fe4b78122196e0453ee956046314b30d57bddde: Status 404 returned error can't find the container with id 2d84b786dce484bcab5a60855fe4b78122196e0453ee956046314b30d57bddde
Sep 10 13:18:16.669359 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:18:16.669333    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e19692c959ce3358ef1453cec7edef.slice/crio-e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 WatchSource:0}: Error finding container e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1: Status 404 returned error can't find the container with id e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1
Sep 10 13:18:16.673061 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.673044    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:18:16.683495 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:16.683455    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:18:16.683601 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:16.683428    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f820c7ef903d  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\",Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:16.673226813 +0000 UTC m=+1.690058764,LastTimestamp:2026-09-10 13:18:16.673226813 +0000 UTC m=+1.690058764,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:16.687310 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:16.687227    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-138-19.ec2.internal.18d3f820c7f046bd  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-138-19.ec2.internal,UID:102348c45e238ec4cfbabd8b1ecec91f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da130d90f0bffd5c5876c94544c91b2a1be0d23ce2747ddfeafe88e03ee7d8d1\",Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:16.673273533 +0000 UTC m=+1.690105500,LastTimestamp:2026-09-10 13:18:16.673273533 +0000 UTC m=+1.690105500,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:16.708529 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:16.708500    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s"
Sep 10 13:18:16.891935 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.891840    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:16.893054 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.893035    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:16.893140 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.893074    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:16.893140 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.893090    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:16.893140 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:16.893125    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:16.911453 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:16.911424    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:17.047046 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:17.047014    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:18:17.299758 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:17.299675    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:17.401644 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:17.401607    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:18:17.404722 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:17.404688    2578 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 10 13:18:17.452310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:17.452240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerStarted","Data":"e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1"}
Sep 10 13:18:17.453619 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:17.453595    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal" event={"ID":"102348c45e238ec4cfbabd8b1ecec91f","Type":"ContainerStarted","Data":"2d84b786dce484bcab5a60855fe4b78122196e0453ee956046314b30d57bddde"}
Sep 10 13:18:18.297503 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.297473    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:18.318398 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.318363    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s"
Sep 10 13:18:18.501734 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.501700    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:18:18.511892 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.511865    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:18.512999 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.512974    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:18.513119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.513016    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:18.513119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.513031    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:18.513119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.513065    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:18.538627 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.538588    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:18.672403 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.672322    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8213e799863  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\" in 1.988s (1.988s including waiting). Image size: 282837706 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.661984355 +0000 UTC m=+3.678816306,LastTimestamp:2026-09-10 13:18:18.661984355 +0000 UTC m=+3.678816306,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:18.694179 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.694101    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-138-19.ec2.internal.18d3f8213fd2db56  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-138-19.ec2.internal,UID:102348c45e238ec4cfbabd8b1ecec91f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da130d90f0bffd5c5876c94544c91b2a1be0d23ce2747ddfeafe88e03ee7d8d1\" in 2.011s (2.011s including waiting). Image size: 321671430 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.684611414 +0000 UTC m=+3.701443379,LastTimestamp:2026-09-10 13:18:18.684611414 +0000 UTC m=+3.701443379,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:18.702643 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.702616    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:18.704064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.704042    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:18.704146 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.704077    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:18.704146 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:18.704088    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:18.712227 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.712191    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:18:18.751854 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.751579    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-138-19.ec2.internal.18d3f82143a69c57  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-138-19.ec2.internal,UID:102348c45e238ec4cfbabd8b1ecec91f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Created container: haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.748820567 +0000 UTC m=+3.765652543,LastTimestamp:2026-09-10 13:18:18.748820567 +0000 UTC m=+3.765652543,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:18.768572 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.768487    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-138-19.ec2.internal.18d3f821441fe6d0  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-138-19.ec2.internal,UID:102348c45e238ec4cfbabd8b1ecec91f,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Started container haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.756769488 +0000 UTC m=+3.773601460,LastTimestamp:2026-09-10 13:18:18.756769488 +0000 UTC m=+3.773601460,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:18.868294 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.868254    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:18:18.953326 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.953242    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214f3bb6ff  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.943141631 +0000 UTC m=+3.959973607,LastTimestamp:2026-09-10 13:18:18.943141631 +0000 UTC m=+3.959973607,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:18.963847 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.963819    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:18:18.963957 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:18.963803    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214fbd2b60  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.951625568 +0000 UTC m=+3.968457520,LastTimestamp:2026-09-10 13:18:18.951625568 +0000 UTC m=+3.968457520,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:19.289979 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.289891    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:19.459275 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.459238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerStarted","Data":"0034d9d221db03f4083405e68bedacf23f9fc9670b9efd68f96cf6735af305e7"}
Sep 10 13:18:19.459275 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.459272    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:19.460305 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.460280    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:19.460431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.460316    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:19.460431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.460328    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:19.460542 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:19.460517    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:19.460881 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.460862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal" event={"ID":"102348c45e238ec4cfbabd8b1ecec91f","Type":"ContainerStarted","Data":"ed6ac3469b04bb3265052d50ea0ca5432389a8baaadaffa037aabc8302985857"}
Sep 10 13:18:19.460959 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.460899    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:19.461622 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.461602    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:19.461688 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.461626    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:19.461688 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:19.461636    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:19.461785 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:19.461773    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:20.306335 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:20.306298    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:20.462624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:20.462595    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:20.463060 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:20.462595    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:20.469052 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:20.469032    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:20.469127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:20.469065    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:20.469127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:20.469076    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:20.469127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:20.469038    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:20.469252 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:20.469143    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:20.469252 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:20.469156    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:20.469337 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:20.469324    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:20.469398 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:20.469385    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:21.298045 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:21.298003    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:21.533330 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:21.533293    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s"
Sep 10 13:18:21.607367 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:21.607276    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:18:21.610407 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:21.610372    2578 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 10 13:18:21.741678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:21.741624    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:21.747298 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:21.747281    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:21.747389 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:21.747316    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:21.747389 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:21.747331    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:21.747389 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:21.747364    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:21.765930 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:21.765899    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:22.298018 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:22.297989    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:22.562647 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:22.562546    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:18:22.802731 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:22.802685    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:22.803825 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:22.803807    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:22.803893 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:22.803848    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:22.803893 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:22.803864    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:23.107592 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:23.107563    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:18:23.299370 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:23.299340    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:24.271062 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:24.271018    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:18:24.297050 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:24.297022    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:24.425759 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:24.425671    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:18:25.298369 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:25.298332    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:25.346358 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:25.346320    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:18:26.299354 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:26.299316    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:27.298892 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:27.298857    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:27.947469 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:27.947435    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:18:28.166786 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:28.166737    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:28.168005 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:28.167987    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:28.168070 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:28.168020    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:28.168070 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:28.168030    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:28.168070 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:28.168056    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:28.188240 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:28.188212    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:28.299967 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:28.299938    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:29.297625 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:29.297589    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:30.297798 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:30.297721    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:30.302921 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:30.302894    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:18:30.305747 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:30.305726    2578 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 10 13:18:30.481706 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:30.481681    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/0.log"
Sep 10 13:18:30.481881 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:30.481716    2578 generic.go:356] "Generic (PLEG): container finished" podID="24e19692c959ce3358ef1453cec7edef" containerID="0034d9d221db03f4083405e68bedacf23f9fc9670b9efd68f96cf6735af305e7" exitCode=1
Sep 10 13:18:30.481881 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:30.481746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerDied","Data":"0034d9d221db03f4083405e68bedacf23f9fc9670b9efd68f96cf6735af305e7"}
Sep 10 13:18:30.481881 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:30.481867    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:30.482662 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:30.482646    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:30.482728 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:30.482675    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:30.482728 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:30.482685    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:30.482875 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:30.482862    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:30.494411 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:30.494325    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f823ff35ec13  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:30.485486611 +0000 UTC m=+15.502318567,LastTimestamp:2026-09-10 13:18:30.485486611 +0000 UTC m=+15.502318567,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:30.572035 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:30.571962    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214f3bb6ff\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214f3bb6ff  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.943141631 +0000 UTC m=+3.959973607,LastTimestamp:2026-09-10 13:18:30.562472233 +0000 UTC m=+15.579304202,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:30.581710 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:30.581637    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214fbd2b60\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214fbd2b60  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.951625568 +0000 UTC m=+3.968457520,LastTimestamp:2026-09-10 13:18:30.570900073 +0000 UTC m=+15.587732037,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:31.301908 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:31.301877    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:31.485029 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:31.484999    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/0.log"
Sep 10 13:18:31.485219 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:31.485050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerStarted","Data":"2cdb55de6543044aad3f470b07307bf626b2983cbd34aa32fc18814637d7f2fd"}
Sep 10 13:18:31.485219 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:31.485181    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:31.486133 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:31.486113    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:31.486205 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:31.486147    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:31.486205 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:31.486157    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:31.486359 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:31.486345    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:32.180726 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:32.180679    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:32.181748 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:32.181732    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:32.181820 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:32.181763    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:32.181820 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:32.181774    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:32.298693 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:32.298662    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:32.649646 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:32.649610    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:18:33.298559 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:33.298524    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:34.297500 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:34.297471    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:34.615708 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:34.615631    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:18:34.959280 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:34.959185    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:18:35.189167 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:35.189127    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:35.190211 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:35.190179    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:35.190315 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:35.190229    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:35.190315 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:35.190240    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:35.190315 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:35.190273    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:35.215176 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:35.215108    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:18:35.215924 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:35.215907    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:35.291448 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:35.291417    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:35.346895 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:35.346861    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:18:36.297514 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:36.297483    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:36.785229 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:36.785179    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:18:37.296271 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:37.296246    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:38.296753 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:38.296725    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:39.297243 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:39.297216    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:40.300120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:40.300089    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:41.298053 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:41.298021    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:41.975717 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:41.975618    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:18:42.216589 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.216550    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:42.217591 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.217561    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:42.217707 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.217601    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:42.217707 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.217612    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:42.217707 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.217638    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:42.235608 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:42.235585    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:42.301502 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.301478    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:42.504273 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.504185    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/1.log"
Sep 10 13:18:42.504605 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.504588    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/0.log"
Sep 10 13:18:42.504691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.504625    2578 generic.go:356] "Generic (PLEG): container finished" podID="24e19692c959ce3358ef1453cec7edef" containerID="2cdb55de6543044aad3f470b07307bf626b2983cbd34aa32fc18814637d7f2fd" exitCode=1
Sep 10 13:18:42.504691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.504664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerDied","Data":"2cdb55de6543044aad3f470b07307bf626b2983cbd34aa32fc18814637d7f2fd"}
Sep 10 13:18:42.504790 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.504757    2578 scope.go:117] "RemoveContainer" containerID="0034d9d221db03f4083405e68bedacf23f9fc9670b9efd68f96cf6735af305e7"
Sep 10 13:18:42.504839 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.504821    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:42.505583 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.505563    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:42.505699 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.505602    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:42.505699 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.505618    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:42.505849 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:42.505834    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:42.505908 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:42.505896    2578 scope.go:117] "RemoveContainer" containerID="0034d9d221db03f4083405e68bedacf23f9fc9670b9efd68f96cf6735af305e7"
Sep 10 13:18:42.513690 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:42.513661    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_0 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: '0034d9d221db03f4083405e68bedacf23f9fc9670b9efd68f96cf6735af305e7'" containerID="0034d9d221db03f4083405e68bedacf23f9fc9670b9efd68f96cf6735af305e7"
Sep 10 13:18:42.513783 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:42.513714    2578 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_0 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: '0034d9d221db03f4083405e68bedacf23f9fc9670b9efd68f96cf6735af305e7'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" logger="UnhandledError"
Sep 10 13:18:42.513899 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:42.513877    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:18:42.525109 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:42.525019    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef),Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:42.513837409 +0000 UTC m=+27.530669382,LastTimestamp:2026-09-10 13:18:42.513837409 +0000 UTC m=+27.530669382,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:43.296559 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:43.296528    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:43.508178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:43.508152    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/1.log"
Sep 10 13:18:44.296239 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:44.296187    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:45.297668 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:45.297641    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:45.347950 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:45.347902    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:18:46.299011 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:46.298979    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:46.869947 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:46.869900    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:18:46.872792 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:46.872770    2578 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 10 13:18:46.872865 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:46.872796    2578 certificate_manager.go:461] "Reached backoff limit, still unable to rotate certs" err="timed out waiting for the condition" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 10 13:18:47.298077 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:47.298046    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:47.491524 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:47.491488    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:18:48.295930 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:48.295903    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:48.984798 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:48.984769    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:18:49.236067 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:49.235992    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:49.237083 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:49.237056    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:49.237209 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:49.237091    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:49.237209 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:49.237104    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:49.237209 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:49.237131    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:49.254393 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:49.254358    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:49.296248 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:49.296211    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:50.283258 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:50.283218    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:18:50.297570 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:50.297541    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:50.411110 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:50.411077    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:50.412073 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:50.412055    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:50.412178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:50.412085    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:50.412178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:50.412095    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:51.301013 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:51.300984    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:51.470781 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:51.470738    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:18:52.298749 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:52.298710    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:53.299583 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:53.299548    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:54.298737 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:54.298703    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:55.298578 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:55.298543    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:55.348894 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:55.348847    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:18:55.445994 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:55.445959    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:55.447148 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:55.447127    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:55.447262 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:55.447167    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:55.447262 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:55.447179    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:55.447399 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:55.447385    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:55.458624 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:55.458534    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f823ff35ec13\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f823ff35ec13  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:30.485486611 +0000 UTC m=+15.502318567,LastTimestamp:2026-09-10 13:18:55.448088993 +0000 UTC m=+40.464920942,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:55.502743 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:55.502714    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:18:55.527838 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:55.527819    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/1.log"
Sep 10 13:18:55.532672 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:55.532604    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214f3bb6ff\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214f3bb6ff  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.943141631 +0000 UTC m=+3.959973607,LastTimestamp:2026-09-10 13:18:55.520404956 +0000 UTC m=+40.537236926,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:55.543621 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:55.543552    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214fbd2b60\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214fbd2b60  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.951625568 +0000 UTC m=+3.968457520,LastTimestamp:2026-09-10 13:18:55.52841412 +0000 UTC m=+40.545246093,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:18:55.996327 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:55.996292    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:18:56.254497 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.254420    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:56.255444 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.255427    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:56.255529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.255458    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:56.255529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.255468    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:56.255529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.255494    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:56.295889 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:56.295862    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:56.295989 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.295916    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:56.531280 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.531183    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/1.log"
Sep 10 13:18:56.531280 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.531252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerStarted","Data":"76812148adbaa8fd8fe316f8f734c6d90f72e209eb14d3ad7daa05ff170cfc73"}
Sep 10 13:18:56.531738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.531432    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:18:56.532540 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.532525    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:18:56.532639 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.532560    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:18:56.532639 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:56.532574    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:18:56.532794 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:18:56.532780    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:18:57.298932 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:57.298899    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:58.300973 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:58.300941    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:18:59.310650 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:18:59.310615    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:00.305884 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:00.305848    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:01.301230 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:01.301182    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:02.300683 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:02.300650    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:03.011475 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:03.011439    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:19:03.296777 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:03.296701    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:03.297597 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:03.297581    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:03.297703 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:03.297611    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:03.297703 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:03.297622    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:03.297703 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:03.297654    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:03.309066 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:03.309038    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:03.316993 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:03.316975    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:04.290996 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:04.290963    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:05.301257 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:05.301227    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:05.349611 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:05.349575    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:19:06.302963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:06.302928    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:07.321831 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.321800    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:07.549605 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.549578    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/2.log"
Sep 10 13:19:07.549933 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.549919    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/1.log"
Sep 10 13:19:07.549993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.549948    2578 generic.go:356] "Generic (PLEG): container finished" podID="24e19692c959ce3358ef1453cec7edef" containerID="76812148adbaa8fd8fe316f8f734c6d90f72e209eb14d3ad7daa05ff170cfc73" exitCode=1
Sep 10 13:19:07.549993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.549977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerDied","Data":"76812148adbaa8fd8fe316f8f734c6d90f72e209eb14d3ad7daa05ff170cfc73"}
Sep 10 13:19:07.550085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.550015    2578 scope.go:117] "RemoveContainer" containerID="2cdb55de6543044aad3f470b07307bf626b2983cbd34aa32fc18814637d7f2fd"
Sep 10 13:19:07.550132 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.550112    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:07.551188 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.550928    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:07.551188 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.550960    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:07.551188 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.550972    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:07.551323 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:07.551224    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:07.551323 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:07.551271    2578 scope.go:117] "RemoveContainer" containerID="2cdb55de6543044aad3f470b07307bf626b2983cbd34aa32fc18814637d7f2fd"
Sep 10 13:19:07.559661 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:07.559638    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_1 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: '2cdb55de6543044aad3f470b07307bf626b2983cbd34aa32fc18814637d7f2fd'" containerID="2cdb55de6543044aad3f470b07307bf626b2983cbd34aa32fc18814637d7f2fd"
Sep 10 13:19:07.559740 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:07.559679    2578 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_1 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: '2cdb55de6543044aad3f470b07307bf626b2983cbd34aa32fc18814637d7f2fd'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" logger="UnhandledError"
Sep 10 13:19:07.559855 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:07.559829    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:19:07.587600 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:07.587456    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef),Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:42.513837409 +0000 UTC m=+27.530669382,LastTimestamp:2026-09-10 13:19:07.559780413 +0000 UTC m=+52.576612379,Count:2,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:19:08.317974 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:08.317946    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:08.552980 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:08.552953    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/2.log"
Sep 10 13:19:09.297617 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:09.297583    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:10.026932 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:10.026898    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:19:10.304791 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:10.304707    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:10.317800 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:10.317777    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:10.319680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:10.319665    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:10.319777 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:10.319694    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:10.319777 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:10.319704    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:10.319777 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:10.319733    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:10.374331 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:10.374298    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:11.305816 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:11.305786    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:12.310348 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:12.310316    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:13.320560 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:13.320530    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:14.300477 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:14.300449    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:14.346019 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:14.345993    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:19:15.300098 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:15.300067    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:15.350619 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:15.350578    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:19:16.333661 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:16.333469    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:17.043992 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:17.043954    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:19:17.317134 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:17.317050    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:17.375376 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:17.375334    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:17.377098 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:17.377078    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:17.377225 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:17.377109    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:17.377225 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:17.377121    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:17.377225 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:17.377148    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:17.412484 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:17.412451    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:17.546517 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:17.546485    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:19:18.301942 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:18.301906    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:18.874446 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:18.874403    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:19:18.879000 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:18.878975    2578 certificate_manager.go:596] "Failed while requesting a signed certificate from the control plane" err="cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User \"system:serviceaccount:openshift-machine-config-operator:node-bootstrapper\" cannot create resource \"certificatesigningrequests\" in API group \"certificates.k8s.io\" at the cluster scope" logger="kubernetes.io/kube-apiserver-client-kubelet.UnhandledError"
Sep 10 13:19:19.308970 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:19.308935    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:20.319785 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:20.319751    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:20.520315 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:20.520281    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:19:21.298348 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:21.298315    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:22.319849 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:22.319813    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:22.445695 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:22.445665    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:22.446665 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:22.446645    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:22.446793 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:22.446681    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:22.446793 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:22.446695    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:22.446942 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:22.446927    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:22.447112 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:22.447093    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:19:22.462285 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:22.462183    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef),Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:42.513837409 +0000 UTC m=+27.530669382,LastTimestamp:2026-09-10 13:19:22.447061206 +0000 UTC m=+67.463893171,Count:3,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:19:23.309416 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:23.309384    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:23.421121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:23.421091    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:23.422156 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:23.422138    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:23.422297 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:23.422167    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:23.422297 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:23.422177    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:24.059165 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:24.059132    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:19:24.303624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:24.303590    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:24.413545 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:24.413481    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:24.414526 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:24.414503    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:24.414637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:24.414539    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:24.414637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:24.414549    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:24.414637 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:24.414573    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:24.456375 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:24.456347    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:25.309097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:25.309065    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:25.351489 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:25.351456    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:19:26.298593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:26.298565    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:27.306890 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:27.306860    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:28.129738 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:28.129711    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:19:28.306020 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:28.305988    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:29.299114 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:29.299079    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:30.304819 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:30.304784    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:31.079109 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:31.079072    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:19:31.300879 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:31.300850    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:31.457053 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:31.456973    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:31.458601 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:31.458584    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:31.458670 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:31.458617    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:31.458670 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:31.458627    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:31.458670 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:31.458657    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:31.479637 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:31.479602    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:32.328962 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:32.328934    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:33.303029 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.302997    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:33.445935 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.445904    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:33.446068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.445909    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:33.446893 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.446879    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:33.446962 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.446908    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:33.446962 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.446917    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:33.447037 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.446879    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:33.447037 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.446979    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:33.447037 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.446989    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:33.447158 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:33.447093    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:33.447223 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:33.447171    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:33.457747 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:33.457670    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f823ff35ec13\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f823ff35ec13  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:30.485486611 +0000 UTC m=+15.502318567,LastTimestamp:2026-09-10 13:19:33.447985688 +0000 UTC m=+78.464817661,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:19:33.538250 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:33.538157    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214f3bb6ff\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214f3bb6ff  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.943141631 +0000 UTC m=+3.959973607,LastTimestamp:2026-09-10 13:19:33.523507441 +0000 UTC m=+78.540339390,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:19:33.553671 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:33.553527    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214fbd2b60\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214fbd2b60  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.951625568 +0000 UTC m=+3.968457520,LastTimestamp:2026-09-10 13:19:33.531684776 +0000 UTC m=+78.548516741,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:19:33.591607 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.591582    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/2.log"
Sep 10 13:19:33.591743 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.591630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerStarted","Data":"6a1f5f7cf8ddc34fb5a0449821637a08e0facabb9c692d0ca60987bf9b1a3e67"}
Sep 10 13:19:33.591783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.591741    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:33.592584 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.592566    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:33.592675 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.592596    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:33.592675 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:33.592606    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:33.592829 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:33.592816    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:34.310385 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:34.310352    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:35.322301 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:35.322273    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:35.352607 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:35.352573    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:19:36.303038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:36.303012    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:37.302969 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:37.302943    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:38.090139 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:38.090112    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:19:38.296874 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:38.296845    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:38.480233 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:38.480191    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:38.481513 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:38.481491    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:38.481613 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:38.481529    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:38.481613 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:38.481540    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:38.481613 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:38.481570    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:38.505629 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:38.505599    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:39.299244 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:39.299190    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:40.300727 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:40.300696    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:41.310121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:41.310090    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:42.299246 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:42.299212    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:43.298517 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:43.298484    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:44.300775 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.300741    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:44.609486 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.609458    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/3.log"
Sep 10 13:19:44.609848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.609831    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/2.log"
Sep 10 13:19:44.609900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.609870    2578 generic.go:356] "Generic (PLEG): container finished" podID="24e19692c959ce3358ef1453cec7edef" containerID="6a1f5f7cf8ddc34fb5a0449821637a08e0facabb9c692d0ca60987bf9b1a3e67" exitCode=1
Sep 10 13:19:44.609934 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.609898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerDied","Data":"6a1f5f7cf8ddc34fb5a0449821637a08e0facabb9c692d0ca60987bf9b1a3e67"}
Sep 10 13:19:44.609967 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.609946    2578 scope.go:117] "RemoveContainer" containerID="76812148adbaa8fd8fe316f8f734c6d90f72e209eb14d3ad7daa05ff170cfc73"
Sep 10 13:19:44.610085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.610072    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:44.611159 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.610882    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:44.611159 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.610918    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:44.611159 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.610935    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:44.611316 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:44.611239    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:44.611316 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:44.611303    2578 scope.go:117] "RemoveContainer" containerID="76812148adbaa8fd8fe316f8f734c6d90f72e209eb14d3ad7daa05ff170cfc73"
Sep 10 13:19:44.619583 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:44.619557    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_2 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: '76812148adbaa8fd8fe316f8f734c6d90f72e209eb14d3ad7daa05ff170cfc73'" containerID="76812148adbaa8fd8fe316f8f734c6d90f72e209eb14d3ad7daa05ff170cfc73"
Sep 10 13:19:44.619660 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:44.619602    2578 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_2 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: '76812148adbaa8fd8fe316f8f734c6d90f72e209eb14d3ad7daa05ff170cfc73'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" logger="UnhandledError"
Sep 10 13:19:44.619747 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:44.619730    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:19:44.633967 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:44.633889    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef),Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:42.513837409 +0000 UTC m=+27.530669382,LastTimestamp:2026-09-10 13:19:44.619697866 +0000 UTC m=+89.636529832,Count:4,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:19:45.148620 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:45.148585    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:19:45.306730 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:45.306704    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:45.353305 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:45.353260    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:19:45.506622 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:45.506595    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:45.507566 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:45.507549    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:45.507653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:45.507578    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:45.507653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:45.507590    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:45.507653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:45.507617    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:45.546611 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:45.546580    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:45.613157 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:45.613135    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/3.log"
Sep 10 13:19:46.300492 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:46.300459    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:47.374498 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:47.374463    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:48.311923 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:48.311894    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:49.369914 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:49.369883    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:50.315423 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:50.315399    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:50.874748 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:50.874708    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 13:19:50.935341 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:50.935318    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 13:19:51.079619 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:51.079588    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d6hpg"
Sep 10 13:19:51.314845 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:51.314818    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:51.454412 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:51.454379    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 13:19:52.167575 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:52.167544    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:19:52.303276 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:52.303254    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:52.547527 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:52.547499    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:52.548565 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:52.548547    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:52.548660 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:52.548581    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:52.548660 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:52.548592    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:52.548660 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:52.548620    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:52.581757 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:52.581734    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:53.301736 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:53.301710    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:53.714889 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:53.714810    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 13:19:54.300934 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:54.300904    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:55.317067 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:55.317035    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:55.354456 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:55.354414    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:19:56.303476 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:56.303448    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:57.314178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:57.314153    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:57.445168 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:57.445140    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:57.446136 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:57.446117    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:57.446224 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:57.446150    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:57.446224 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:57.446160    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:57.446366 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:57.446354    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:57.446502 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:57.446486    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:19:57.469013 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:57.468865    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef),Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:42.513837409 +0000 UTC m=+27.530669382,LastTimestamp:2026-09-10 13:19:57.446461025 +0000 UTC m=+102.463292990,Count:5,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:19:58.310325 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:58.310295    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:59.180536 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:59.180498    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:19:59.310866 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:59.310834    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:19:59.582502 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:59.582474    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:19:59.583523 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:59.583504    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:19:59.583653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:59.583535    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:19:59.583653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:59.583564    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:19:59.583653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:19:59.583590    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:19:59.617836 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:19:59.617806    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:00.300849 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:00.300814    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:01.317529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:01.317502    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:02.305399 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:02.305364    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:03.305358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:03.305326    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:04.309902 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:04.309869    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:05.315731 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:05.315696    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:05.355262 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:05.355234    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:20:06.202518 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:06.202481    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:20:06.321674 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:06.321645    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:06.618347 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:06.618311    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:20:06.619349 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:06.619332    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:20:06.619438 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:06.619362    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:20:06.619438 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:06.619376    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:20:06.619438 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:06.619405    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:06.664142 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:06.664111    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:07.305042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:07.305006    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:07.341182 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:07.341150    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Sep 10 13:20:08.321185 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:08.321153    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:09.308805 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:09.308770    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:10.304750 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:10.304691    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:11.304010 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:11.303983    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:12.314512 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:12.314489    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:12.445487 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:12.445455    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:20:12.446435 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:12.446419    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:20:12.446510 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:12.446448    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:20:12.446510 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:12.446459    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:20:12.446671 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:12.446659    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:12.446809 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:12.446793    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 40s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:20:12.456805 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:12.456729    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f826cc27fd61  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:BackOff,Message:Back-off restarting failed container setup in pod kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef),Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:42.513837409 +0000 UTC m=+27.530669382,LastTimestamp:2026-09-10 13:20:12.446767437 +0000 UTC m=+117.463599399,Count:6,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:20:13.224717 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:13.224681    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:20:13.297791 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:13.297764    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:13.664688 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:13.664661    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:20:13.665596 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:13.665580    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:20:13.665686 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:13.665610    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:20:13.665686 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:13.665619    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:20:13.665686 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:13.665647    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:13.702179 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:13.702153    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:14.300904 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:14.300873    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:15.300867 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:15.300843    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:15.355679 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:15.355638    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:20:16.300404 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:16.300372    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:17.314768 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:17.314731    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:18.301971 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:18.301944    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:19.301038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:19.301010    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:20.254683 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:20.254651    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s"
Sep 10 13:20:20.311127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:20.311100    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:20.702935 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:20.702843    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:20:20.704558 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:20.704533    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:20:20.704668 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:20.704567    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:20:20.704668 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:20.704577    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:20:20.704668 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:20.704605    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:20.742937 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:20.742908    2578 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:21.304461 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:21.304428    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:22.309034 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:22.309003    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:22.902716 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:22.902681    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 13:20:23.301553 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:23.301528    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:24.319060 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:24.319028    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:25.297456 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.297428    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 13:20:25.356017 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:25.355981    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:20:25.445462 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.445425    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:20:25.448455 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.448440    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:20:25.448528 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.448472    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:20:25.448528 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.448481    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:20:25.448684 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:25.448673    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:25.462284 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:25.462179    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f823ff35ec13\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f823ff35ec13  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cec847b1f08cae0315bd44fe477d725a0180b78ab0ad844de3b218bf2dde0bb2\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:30.485486611 +0000 UTC m=+15.502318567,LastTimestamp:2026-09-10 13:20:25.449429611 +0000 UTC m=+130.466261580,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:20:25.528711 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:25.528639    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214f3bb6ff\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214f3bb6ff  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.943141631 +0000 UTC m=+3.959973607,LastTimestamp:2026-09-10 13:20:25.517308299 +0000 UTC m=+130.534140261,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:20:25.538997 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:25.538930    2578 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214fbd2b60\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal.18d3f8214fbd2b60  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal,UID:24e19692c959ce3358ef1453cec7edef,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-138-19.ec2.internal,},FirstTimestamp:2026-09-10 13:18:18.951625568 +0000 UTC m=+3.968457520,LastTimestamp:2026-09-10 13:20:25.525112418 +0000 UTC m=+130.541944380,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-19.ec2.internal,}"
Sep 10 13:20:25.673164 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.673093    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/3.log"
Sep 10 13:20:25.673164 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.673141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerStarted","Data":"f227eae44fb7abb0d51682b3ca9423f8b89afb8096958572a6b3adbfcb20cf46"}
Sep 10 13:20:25.673334 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.673265    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:20:25.674016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.674002    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:20:25.674066 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.674027    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:20:25.674066 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:25.674037    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:20:25.674242 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:25.674230    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:26.111320 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:26.111289    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d6hpg"
Sep 10 13:20:26.236361 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:26.236313    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 13:20:26.353313 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:26.353281    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:26.394286 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:26.394212    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:26.501552 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:26.501527    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:26.798383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:26.798356    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:26.798383 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:26.798381    2578 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:26.869932 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:26.869906    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:26.925752 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:26.925729    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:26.997245 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:26.997216    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:27.114763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.114679    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 13:15:26 +0000 UTC" deadline="2028-03-02 02:25:46.416545901 +0000 UTC"
Sep 10 13:20:27.114763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.114713    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12925h5m19.301836954s"
Sep 10 13:20:27.269093 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.269057    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:27.269093 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:27.269083    2578 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:27.269283 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:27.269125    2578 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:27.532382 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.532353    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:27.578386 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.578357    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:27.675093 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.675063    2578 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-138-19.ec2.internal" not found
Sep 10 13:20:27.743539 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.743508    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:20:27.744467 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.744449    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:20:27.744558 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.744479    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:20:27.744558 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.744488    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:20:27.744558 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.744519    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:27.766705 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:27.766682    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:27.766771 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:27.766710    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-19.ec2.internal\": node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:20:27.831179 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:27.831102    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:20:27.931609 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:27.931560    2578 kubelet_node_status.go:448] "Node not becoming ready in time after startup"
Sep 10 13:20:30.370299 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:30.370259    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:20:31.906973 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:31.906928    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:20:31.909354 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:31.909336    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:20:31.909428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:31.909368    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:20:31.909428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:31.909378    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:20:35.356312 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:35.356271    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-19.ec2.internal\" not found"
Sep 10 13:20:35.371004 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:35.370974    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:20:36.691539 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.691514    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/4.log"
Sep 10 13:20:36.691938 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.691892    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/3.log"
Sep 10 13:20:36.691938 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.691922    2578 generic.go:356] "Generic (PLEG): container finished" podID="24e19692c959ce3358ef1453cec7edef" containerID="f227eae44fb7abb0d51682b3ca9423f8b89afb8096958572a6b3adbfcb20cf46" exitCode=1
Sep 10 13:20:36.692005 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.691950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerDied","Data":"f227eae44fb7abb0d51682b3ca9423f8b89afb8096958572a6b3adbfcb20cf46"}
Sep 10 13:20:36.692041 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.692019    2578 scope.go:117] "RemoveContainer" containerID="6a1f5f7cf8ddc34fb5a0449821637a08e0facabb9c692d0ca60987bf9b1a3e67"
Sep 10 13:20:36.692104 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.692090    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 13:20:36.693249 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.692967    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 13:20:36.693249 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.693041    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 13:20:36.693249 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.693057    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeHasSufficientPID"
Sep 10 13:20:36.693415 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:36.693307    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-19.ec2.internal\" not found" node="ip-10-0-138-19.ec2.internal"
Sep 10 13:20:36.693415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:36.693366    2578 scope.go:117] "RemoveContainer" containerID="6a1f5f7cf8ddc34fb5a0449821637a08e0facabb9c692d0ca60987bf9b1a3e67"
Sep 10 13:20:36.701102 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:36.701068    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_3 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: '6a1f5f7cf8ddc34fb5a0449821637a08e0facabb9c692d0ca60987bf9b1a3e67'" containerID="6a1f5f7cf8ddc34fb5a0449821637a08e0facabb9c692d0ca60987bf9b1a3e67"
Sep 10 13:20:36.701225 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:36.701118    2578 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_3 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: '6a1f5f7cf8ddc34fb5a0449821637a08e0facabb9c692d0ca60987bf9b1a3e67'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" logger="UnhandledError"
Sep 10 13:20:36.701324 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:36.701304    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:20:37.635793 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:37.635759    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:20:37.694826 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:37.694801    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/4.log"
Sep 10 13:20:37.695177 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:37.694865    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal"
Sep 10 13:20:37.715466 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:37.715442    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 13:20:37.717892 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:37.717879    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal"
Sep 10 13:20:37.732167 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:37.732146    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 13:20:38.348305 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.348259    2578 apiserver.go:52] "Watching apiserver"
Sep 10 13:20:38.357332 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.357300    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 13:20:38.357635 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.357614    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-62tzd","kube-system/konnectivity-agent-z52mw","kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h","openshift-network-diagnostics/network-check-target-llj8p","openshift-network-operator/iptables-alerter-xq466","openshift-cluster-node-tuning-operator/tuned-8b7fp","openshift-image-registry/node-ca-64j9z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal","openshift-multus/multus-additional-cni-plugins-r672x","openshift-multus/multus-s69g5","openshift-multus/network-metrics-daemon-6b7mv","openshift-ovn-kubernetes/ovnkube-node-6pv7m"]
Sep 10 13:20:38.361526 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.361508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.363730 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.363709    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:38.363844 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.363797    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:38.364956 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.364941    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rg9tv\""
Sep 10 13:20:38.365062 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.365045    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 13:20:38.365165 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.365151    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 13:20:38.365476 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.365460    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 13:20:38.365797 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.365784    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.367922 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.367906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.370029 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.370011    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:38.370119 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.370074    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:38.372295 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.372243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.374411 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.374396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:38.374501 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.374466    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:38.376732 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.376716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.376909 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.376890    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 13:20:38.377135 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.377118    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 13:20:38.377259 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.377242    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 13:20:38.377342 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.377325    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qmjpn\""
Sep 10 13:20:38.377393 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.377340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 13:20:38.377449 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.377399    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 13:20:38.377605 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.377463    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 13:20:38.378001 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.377982    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 13:20:38.378080 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.377991    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 13:20:38.378151 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.378134    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 13:20:38.378274 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.378258    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lmg57\""
Sep 10 13:20:38.379010 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.378977    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 13:20:38.379438 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.379417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.382721 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.382702    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:38.385230 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.385210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.385526 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.385452    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:20:38.394499 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.394480    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 13:20:38.394649 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.394501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 13:20:38.394725 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.394501    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mzt68\""
Sep 10 13:20:38.394725 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.394531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 13:20:38.394725 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.394603    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 13:20:38.394870 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.394638    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 13:20:38.394870 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.394781    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 13:20:38.395059 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.394905    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7ztgj\""
Sep 10 13:20:38.395766 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.395747    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 13:20:38.395766 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.395764    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9hn4m\""
Sep 10 13:20:38.395900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.395777    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 13:20:38.395900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.395783    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 13:20:38.395900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.395790    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 13:20:38.396039 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.395916    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 13:20:38.396488 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.396469    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 13:20:38.396550 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.396477    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 13:20:38.397263 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.397248    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vx2vm\""
Sep 10 13:20:38.405651 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.405633    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 13:20:38.406234 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.406217    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-czbtf\""
Sep 10 13:20:38.409478 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.409460    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 13:20:38.422203 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-modprobe-d\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.422294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-script-lib\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.422294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422231    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-sys-fs\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.422294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422268    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-cni-binary-copy\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.422294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422285    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2vnf\" (UniqueName: \"kubernetes.io/projected/7e2ddbb0-f6a4-4610-8df2-55ff53f41368-kube-api-access-r2vnf\") pod \"iptables-alerter-xq466\" (UID: \"7e2ddbb0-f6a4-4610-8df2-55ff53f41368\") " pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.422458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-kubelet-config\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:38.422458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422345    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-dbus\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:38.422458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-registration-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.422458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-run-netns\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.422458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-var-lib-cni-multus\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.422458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422422    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-run-multus-certs\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.422458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-sysctl-conf\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.422458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-socket-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-etc-selinux\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60c25acb-86d9-426a-9685-1818f7c32000-ovn-node-metrics-cert\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-sysconfig\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-tmp\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422533    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/104356b6-35e4-472f-b715-aca908204fb4-system-cni-dir\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-netns\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-ovn\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-netd\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lnhh\" (UniqueName: \"kubernetes.io/projected/60c25acb-86d9-426a-9685-1818f7c32000-kube-api-access-7lnhh\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-run-k8s-cni-cncf-io\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-var-lib-cni-bin\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.422673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2ddbb0-f6a4-4610-8df2-55ff53f41368-host-slash\") pod \"iptables-alerter-xq466\" (UID: \"7e2ddbb0-f6a4-4610-8df2-55ff53f41368\") " pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28616fa8-74fb-434f-8238-4246116c85b2-host\") pod \"node-ca-64j9z\" (UID: \"28616fa8-74fb-434f-8238-4246116c85b2\") " pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-daemon-config\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-host\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/104356b6-35e4-472f-b715-aca908204fb4-cnibin\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-systemd\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-config\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-hostroot\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbe2e10-4ae2-4914-9e54-fa46bdc66b94-agent-certs\") pod \"konnectivity-agent-z52mw\" (UID: \"8fbe2e10-4ae2-4914-9e54-fa46bdc66b94\") " pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-systemd-units\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422919    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-slash\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-kubernetes\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.423002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.422991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-sysctl-d\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423009    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423036    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/104356b6-35e4-472f-b715-aca908204fb4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5sw\" (UniqueName: \"kubernetes.io/projected/104356b6-35e4-472f-b715-aca908204fb4-kube-api-access-bp5sw\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7fh\" (UniqueName: \"kubernetes.io/projected/28616fa8-74fb-434f-8238-4246116c85b2-kube-api-access-7d7fh\") pod \"node-ca-64j9z\" (UID: \"28616fa8-74fb-434f-8238-4246116c85b2\") " pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s246d\" (UniqueName: \"kubernetes.io/projected/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-kube-api-access-s246d\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-systemd\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/104356b6-35e4-472f-b715-aca908204fb4-cni-binary-copy\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/104356b6-35e4-472f-b715-aca908204fb4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-openvswitch\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-env-overrides\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28616fa8-74fb-434f-8238-4246116c85b2-serviceca\") pod \"node-ca-64j9z\" (UID: \"28616fa8-74fb-434f-8238-4246116c85b2\") " pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-var-lib-kubelet\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-conf-dir\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-lib-modules\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423265    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.423410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423280    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-node-log\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423295    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-bin\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-system-cni-dir\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbe2e10-4ae2-4914-9e54-fa46bdc66b94-konnectivity-ca\") pod \"konnectivity-agent-z52mw\" (UID: \"8fbe2e10-4ae2-4914-9e54-fa46bdc66b94\") " pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-device-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-kubelet\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423407    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-etc-openvswitch\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-cnibin\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-etc-kubernetes\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423446    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-var-lib-openvswitch\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-socket-dir-parent\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-sys\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-var-lib-kubelet\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-tuned\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423549    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7cz8\" (UniqueName: \"kubernetes.io/projected/68b95818-d562-46cd-808a-c7a2df57ecdf-kube-api-access-h7cz8\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.423848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-log-socket\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.424300 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423578    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-ovn-kubernetes\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.424300 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e2ddbb0-f6a4-4610-8df2-55ff53f41368-iptables-alerter-script\") pod \"iptables-alerter-xq466\" (UID: \"7e2ddbb0-f6a4-4610-8df2-55ff53f41368\") " pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.424300 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-run\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.424300 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khj8g\" (UniqueName: \"kubernetes.io/projected/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-kube-api-access-khj8g\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.424300 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-cni-dir\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.424300 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8286\" (UniqueName: \"kubernetes.io/projected/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-kube-api-access-d8286\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:38.424300 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.423672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/104356b6-35e4-472f-b715-aca908204fb4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.486531 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.486500    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nlkw5"
Sep 10 13:20:38.501160 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.501138    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nlkw5"
Sep 10 13:20:38.524083 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28616fa8-74fb-434f-8238-4246116c85b2-host\") pod \"node-ca-64j9z\" (UID: \"28616fa8-74fb-434f-8238-4246116c85b2\") " pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.524187 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-daemon-config\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.524187 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-host\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.524187 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/104356b6-35e4-472f-b715-aca908204fb4-cnibin\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.524187 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-systemd\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.524187 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524167    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28616fa8-74fb-434f-8238-4246116c85b2-host\") pod \"node-ca-64j9z\" (UID: \"28616fa8-74fb-434f-8238-4246116c85b2\") " pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.524187 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-config\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-host\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-hostroot\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbe2e10-4ae2-4914-9e54-fa46bdc66b94-agent-certs\") pod \"konnectivity-agent-z52mw\" (UID: \"8fbe2e10-4ae2-4914-9e54-fa46bdc66b94\") " pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-systemd\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/104356b6-35e4-472f-b715-aca908204fb4-cnibin\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524261    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-systemd-units\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-slash\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-hostroot\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-kubernetes\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-sysctl-d\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.524415 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.524403    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/104356b6-35e4-472f-b715-aca908204fb4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.524505    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs podName:25ca8aaa-ce03-426c-bc78-66e0fbebc64c nodeName:}" failed. No retries permitted until 2026-09-10 13:20:39.024454962 +0000 UTC m=+144.041286925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs") pod "network-metrics-daemon-6b7mv" (UID: "25ca8aaa-ce03-426c-bc78-66e0fbebc64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-slash\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524535    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5sw\" (UniqueName: \"kubernetes.io/projected/104356b6-35e4-472f-b715-aca908204fb4-kube-api-access-bp5sw\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7fh\" (UniqueName: \"kubernetes.io/projected/28616fa8-74fb-434f-8238-4246116c85b2-kube-api-access-7d7fh\") pod \"node-ca-64j9z\" (UID: \"28616fa8-74fb-434f-8238-4246116c85b2\") " pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s246d\" (UniqueName: \"kubernetes.io/projected/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-kube-api-access-s246d\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-systemd\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/104356b6-35e4-472f-b715-aca908204fb4-cni-binary-copy\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/104356b6-35e4-472f-b715-aca908204fb4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-sysctl-d\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-openvswitch\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-kubernetes\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-env-overrides\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28616fa8-74fb-434f-8238-4246116c85b2-serviceca\") pod \"node-ca-64j9z\" (UID: \"28616fa8-74fb-434f-8238-4246116c85b2\") " pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.524861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-var-lib-kubelet\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-daemon-config\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-conf-dir\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-lib-modules\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.524833    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-node-log\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.524892    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret podName:c9e7452e-8ac7-40b2-a170-fb36f5f4cff9 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:39.024867207 +0000 UTC m=+144.041699156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret") pod "global-pull-secret-syncer-62tzd" (UID: "c9e7452e-8ac7-40b2-a170-fb36f5f4cff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-node-log\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524903    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-openvswitch\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-bin\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/104356b6-35e4-472f-b715-aca908204fb4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524956    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-system-cni-dir\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbe2e10-4ae2-4914-9e54-fa46bdc66b94-konnectivity-ca\") pod \"konnectivity-agent-z52mw\" (UID: \"8fbe2e10-4ae2-4914-9e54-fa46bdc66b94\") " pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.524997    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-system-cni-dir\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525037    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-bin\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.525407 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-device-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525047    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-kubelet\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-conf-dir\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525093    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-etc-openvswitch\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-cnibin\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525137    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/104356b6-35e4-472f-b715-aca908204fb4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-etc-kubernetes\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-var-lib-openvswitch\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-socket-dir-parent\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-sys\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525272    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-lib-modules\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525286    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-var-lib-kubelet\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-tuned\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-systemd\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7cz8\" (UniqueName: \"kubernetes.io/projected/68b95818-d562-46cd-808a-c7a2df57ecdf-kube-api-access-h7cz8\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-log-socket\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-cnibin\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.526064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-ovn-kubernetes\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e2ddbb0-f6a4-4610-8df2-55ff53f41368-iptables-alerter-script\") pod \"iptables-alerter-xq466\" (UID: \"7e2ddbb0-f6a4-4610-8df2-55ff53f41368\") " pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-systemd-units\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-run\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525454    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-var-lib-kubelet\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-var-lib-kubelet\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khj8g\" (UniqueName: \"kubernetes.io/projected/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-kube-api-access-khj8g\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-cni-dir\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-etc-kubernetes\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8286\" (UniqueName: \"kubernetes.io/projected/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-kube-api-access-d8286\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/104356b6-35e4-472f-b715-aca908204fb4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-socket-dir-parent\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-modprobe-d\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525605    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-sys\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-script-lib\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-sys-fs\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.526802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-log-socket\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-env-overrides\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-cni-binary-copy\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2vnf\" (UniqueName: \"kubernetes.io/projected/7e2ddbb0-f6a4-4610-8df2-55ff53f41368-kube-api-access-r2vnf\") pod \"iptables-alerter-xq466\" (UID: \"7e2ddbb0-f6a4-4610-8df2-55ff53f41368\") " pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-ovn-kubernetes\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-kubelet-config\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/104356b6-35e4-472f-b715-aca908204fb4-cni-binary-copy\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-dbus\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525770    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-multus-cni-dir\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-registration-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-kubelet-config\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-run-netns\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-var-lib-cni-multus\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-registration-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28616fa8-74fb-434f-8238-4246116c85b2-serviceca\") pod \"node-ca-64j9z\" (UID: \"28616fa8-74fb-434f-8238-4246116c85b2\") " pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525860    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-dbus\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-run-multus-certs\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.527581 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525881    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8fbe2e10-4ae2-4914-9e54-fa46bdc66b94-konnectivity-ca\") pod \"konnectivity-agent-z52mw\" (UID: \"8fbe2e10-4ae2-4914-9e54-fa46bdc66b94\") " pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-sys-fs\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-device-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525971    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-var-lib-cni-multus\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-kubelet\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/104356b6-35e4-472f-b715-aca908204fb4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-etc-openvswitch\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-sysctl-conf\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-run-netns\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.525659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-var-lib-openvswitch\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7e2ddbb0-f6a4-4610-8df2-55ff53f41368-iptables-alerter-script\") pod \"iptables-alerter-xq466\" (UID: \"7e2ddbb0-f6a4-4610-8df2-55ff53f41368\") " pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-socket-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-socket-dir\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-etc-selinux\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-run-multus-certs\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60c25acb-86d9-426a-9685-1818f7c32000-ovn-node-metrics-cert\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-sysconfig\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.528383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-tmp\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-modprobe-d\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/104356b6-35e4-472f-b715-aca908204fb4-system-cni-dir\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-netns\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-run\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-ovn\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-netd\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-sysconfig\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lnhh\" (UniqueName: \"kubernetes.io/projected/60c25acb-86d9-426a-9685-1818f7c32000-kube-api-access-7lnhh\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-run-k8s-cni-cncf-io\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68b95818-d562-46cd-808a-c7a2df57ecdf-etc-selinux\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526450    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-var-lib-cni-bin\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-config\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/104356b6-35e4-472f-b715-aca908204fb4-system-cni-dir\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526479    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2ddbb0-f6a4-4610-8df2-55ff53f41368-host-slash\") pod \"iptables-alerter-xq466\" (UID: \"7e2ddbb0-f6a4-4610-8df2-55ff53f41368\") " pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526507    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e2ddbb0-f6a4-4610-8df2-55ff53f41368-host-slash\") pod \"iptables-alerter-xq466\" (UID: \"7e2ddbb0-f6a4-4610-8df2-55ff53f41368\") " pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-sysctl-conf\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.529108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526516    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-cni-binary-copy\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-ovn\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-var-lib-cni-bin\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526588    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-netns\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526624    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-netd\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-host-run-k8s-cni-cncf-io\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526668    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.526745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-script-lib\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.529152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-etc-tuned\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.529233    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-tmp\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.529370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60c25acb-86d9-426a-9685-1818f7c32000-ovn-node-metrics-cert\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.529640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.529558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8fbe2e10-4ae2-4914-9e54-fa46bdc66b94-agent-certs\") pod \"konnectivity-agent-z52mw\" (UID: \"8fbe2e10-4ae2-4914-9e54-fa46bdc66b94\") " pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:38.552367 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.552323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5sw\" (UniqueName: \"kubernetes.io/projected/104356b6-35e4-472f-b715-aca908204fb4-kube-api-access-bp5sw\") pod \"multus-additional-cni-plugins-r672x\" (UID: \"104356b6-35e4-472f-b715-aca908204fb4\") " pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.552845 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.552827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s246d\" (UniqueName: \"kubernetes.io/projected/5bf7fbef-791b-4e8c-8c6d-baed2bc964ee-kube-api-access-s246d\") pod \"multus-s69g5\" (UID: \"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee\") " pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.553078 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.553056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lnhh\" (UniqueName: \"kubernetes.io/projected/60c25acb-86d9-426a-9685-1818f7c32000-kube-api-access-7lnhh\") pod \"ovnkube-node-6pv7m\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") " pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.556734 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.556716    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:20:38.556734 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.556737    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:20:38.556846 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.556748    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pd4dk for pod openshift-network-diagnostics/network-check-target-llj8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:38.556846 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:38.556804    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk podName:f64e39f7-de8e-43ea-987a-02c4811d7050 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:39.056791217 +0000 UTC m=+144.073623209 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pd4dk" (UniqueName: "kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk") pod "network-check-target-llj8p" (UID: "f64e39f7-de8e-43ea-987a-02c4811d7050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:38.558984 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.558956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2vnf\" (UniqueName: \"kubernetes.io/projected/7e2ddbb0-f6a4-4610-8df2-55ff53f41368-kube-api-access-r2vnf\") pod \"iptables-alerter-xq466\" (UID: \"7e2ddbb0-f6a4-4610-8df2-55ff53f41368\") " pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.559706 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.559674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khj8g\" (UniqueName: \"kubernetes.io/projected/7cca75e3-1d89-4758-9cd2-ad515da5ffe4-kube-api-access-khj8g\") pod \"tuned-8b7fp\" (UID: \"7cca75e3-1d89-4758-9cd2-ad515da5ffe4\") " pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.560007 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.559989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8286\" (UniqueName: \"kubernetes.io/projected/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-kube-api-access-d8286\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:38.560070 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.560048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7cz8\" (UniqueName: \"kubernetes.io/projected/68b95818-d562-46cd-808a-c7a2df57ecdf-kube-api-access-h7cz8\") pod \"aws-ebs-csi-driver-node-j4n6h\" (UID: \"68b95818-d562-46cd-808a-c7a2df57ecdf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.560618 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.560602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7fh\" (UniqueName: \"kubernetes.io/projected/28616fa8-74fb-434f-8238-4246116c85b2-kube-api-access-7d7fh\") pod \"node-ca-64j9z\" (UID: \"28616fa8-74fb-434f-8238-4246116c85b2\") " pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.670258 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.670172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-64j9z"
Sep 10 13:20:38.675626 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.675603    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s69g5"
Sep 10 13:20:38.681388 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.681364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8b7fp"
Sep 10 13:20:38.682756 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:20:38.682727    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28616fa8_74fb_434f_8238_4246116c85b2.slice/crio-107804f4c9a6cbc14e15afd42a0731b9c851ef9b166fe7c9748b320e4c8516f3 WatchSource:0}: Error finding container 107804f4c9a6cbc14e15afd42a0731b9c851ef9b166fe7c9748b320e4c8516f3: Status 404 returned error can't find the container with id 107804f4c9a6cbc14e15afd42a0731b9c851ef9b166fe7c9748b320e4c8516f3
Sep 10 13:20:38.687179 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.687164    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xq466"
Sep 10 13:20:38.690229 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:20:38.690189    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cca75e3_1d89_4758_9cd2_ad515da5ffe4.slice/crio-fce0333ca5cbea8c324ce22b6ba77aefb3e20669895602ce0a52a67470bbec34 WatchSource:0}: Error finding container fce0333ca5cbea8c324ce22b6ba77aefb3e20669895602ce0a52a67470bbec34: Status 404 returned error can't find the container with id fce0333ca5cbea8c324ce22b6ba77aefb3e20669895602ce0a52a67470bbec34
Sep 10 13:20:38.692122 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.692097    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h"
Sep 10 13:20:38.696568 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:20:38.696542    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2ddbb0_f6a4_4610_8df2_55ff53f41368.slice/crio-c5a31ef01cd9efd3e5e2557b6ecca9d89959609e5b00591fd8e4d79b650d4f06 WatchSource:0}: Error finding container c5a31ef01cd9efd3e5e2557b6ecca9d89959609e5b00591fd8e4d79b650d4f06: Status 404 returned error can't find the container with id c5a31ef01cd9efd3e5e2557b6ecca9d89959609e5b00591fd8e4d79b650d4f06
Sep 10 13:20:38.697727 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.697706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r672x"
Sep 10 13:20:38.698133 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.698092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8b7fp" event={"ID":"7cca75e3-1d89-4758-9cd2-ad515da5ffe4","Type":"ContainerStarted","Data":"fce0333ca5cbea8c324ce22b6ba77aefb3e20669895602ce0a52a67470bbec34"}
Sep 10 13:20:38.699281 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.699231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s69g5" event={"ID":"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee","Type":"ContainerStarted","Data":"1c61032b891b0f526d1a09990d9294de9d3c6eae02058e057bdc4e8c1d34225d"}
Sep 10 13:20:38.700515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.700302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-64j9z" event={"ID":"28616fa8-74fb-434f-8238-4246116c85b2","Type":"ContainerStarted","Data":"107804f4c9a6cbc14e15afd42a0731b9c851ef9b166fe7c9748b320e4c8516f3"}
Sep 10 13:20:38.701659 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:20:38.701639    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b95818_d562_46cd_808a_c7a2df57ecdf.slice/crio-ba35cdc0288e04b5c9605c7f115e1e8d880c44a2913b8f6f18664bf212e7480b WatchSource:0}: Error finding container ba35cdc0288e04b5c9605c7f115e1e8d880c44a2913b8f6f18664bf212e7480b: Status 404 returned error can't find the container with id ba35cdc0288e04b5c9605c7f115e1e8d880c44a2913b8f6f18664bf212e7480b
Sep 10 13:20:38.702979 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.702962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:38.706564 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:20:38.706540    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod104356b6_35e4_472f_b715_aca908204fb4.slice/crio-7b6eb9aba31d21788324db30cf4ec38e5511fe5a08de3795498f813563035349 WatchSource:0}: Error finding container 7b6eb9aba31d21788324db30cf4ec38e5511fe5a08de3795498f813563035349: Status 404 returned error can't find the container with id 7b6eb9aba31d21788324db30cf4ec38e5511fe5a08de3795498f813563035349
Sep 10 13:20:38.707535 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.707513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:20:38.716803 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:20:38.716754    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c25acb_86d9_426a_9685_1818f7c32000.slice/crio-311cc1b7a6735685758cf5563d9dc7d236b7530cd9c241205da18820ddb920c0 WatchSource:0}: Error finding container 311cc1b7a6735685758cf5563d9dc7d236b7530cd9c241205da18820ddb920c0: Status 404 returned error can't find the container with id 311cc1b7a6735685758cf5563d9dc7d236b7530cd9c241205da18820ddb920c0
Sep 10 13:20:38.725956 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.725902    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-19.ec2.internal" podStartSLOduration=1.725886845 podStartE2EDuration="1.725886845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:20:38.671557075 +0000 UTC m=+143.688389044" watchObservedRunningTime="2026-09-10 13:20:38.725886845 +0000 UTC m=+143.742718840"
Sep 10 13:20:38.726678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:38.726658    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-6pv7m"]
Sep 10 13:20:39.029418 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.029388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:39.029418 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.029423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:39.029619 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:39.029535    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:39.029619 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:39.029582    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret podName:c9e7452e-8ac7-40b2-a170-fb36f5f4cff9 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:40.0295706 +0000 UTC m=+145.046402550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret") pod "global-pull-secret-syncer-62tzd" (UID: "c9e7452e-8ac7-40b2-a170-fb36f5f4cff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:39.029619 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:39.029536    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:39.029717 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:39.029659    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs podName:25ca8aaa-ce03-426c-bc78-66e0fbebc64c nodeName:}" failed. No retries permitted until 2026-09-10 13:20:40.029646272 +0000 UTC m=+145.046478236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs") pod "network-metrics-daemon-6b7mv" (UID: "25ca8aaa-ce03-426c-bc78-66e0fbebc64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:39.130415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.130373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:39.130596 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:39.130554    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:20:39.130596 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:39.130579    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:20:39.130596 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:39.130592    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pd4dk for pod openshift-network-diagnostics/network-check-target-llj8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:39.130738 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:39.130660    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk podName:f64e39f7-de8e-43ea-987a-02c4811d7050 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:40.130642044 +0000 UTC m=+145.147474013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pd4dk" (UniqueName: "kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk") pod "network-check-target-llj8p" (UID: "f64e39f7-de8e-43ea-987a-02c4811d7050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:39.446521 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.446436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:39.446712 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:39.446578    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:39.504691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.504648    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:15:38 +0000 UTC" deadline="2028-02-29 09:04:57.898388565 +0000 UTC"
Sep 10 13:20:39.504691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.504689    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12883h44m18.393704007s"
Sep 10 13:20:39.708420 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.708334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerStarted","Data":"311cc1b7a6735685758cf5563d9dc7d236b7530cd9c241205da18820ddb920c0"}
Sep 10 13:20:39.715596 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.715562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z52mw" event={"ID":"8fbe2e10-4ae2-4914-9e54-fa46bdc66b94","Type":"ContainerStarted","Data":"8e00dfbd1634b2467128aacc4503e2d4327872168175fb39f6ff879bae72830a"}
Sep 10 13:20:39.722137 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.722104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerStarted","Data":"7b6eb9aba31d21788324db30cf4ec38e5511fe5a08de3795498f813563035349"}
Sep 10 13:20:39.726162 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.726131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h" event={"ID":"68b95818-d562-46cd-808a-c7a2df57ecdf","Type":"ContainerStarted","Data":"ba35cdc0288e04b5c9605c7f115e1e8d880c44a2913b8f6f18664bf212e7480b"}
Sep 10 13:20:39.728967 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:39.728940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xq466" event={"ID":"7e2ddbb0-f6a4-4610-8df2-55ff53f41368","Type":"ContainerStarted","Data":"c5a31ef01cd9efd3e5e2557b6ecca9d89959609e5b00591fd8e4d79b650d4f06"}
Sep 10 13:20:40.037698 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:40.037659    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:40.037890 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:40.037725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:40.037890 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.037850    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:40.038005 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.037910    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret podName:c9e7452e-8ac7-40b2-a170-fb36f5f4cff9 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:42.037892651 +0000 UTC m=+147.054724614 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret") pod "global-pull-secret-syncer-62tzd" (UID: "c9e7452e-8ac7-40b2-a170-fb36f5f4cff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:40.038451 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.038349    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:40.038451 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.038412    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs podName:25ca8aaa-ce03-426c-bc78-66e0fbebc64c nodeName:}" failed. No retries permitted until 2026-09-10 13:20:42.038387401 +0000 UTC m=+147.055219366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs") pod "network-metrics-daemon-6b7mv" (UID: "25ca8aaa-ce03-426c-bc78-66e0fbebc64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:40.138748 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:40.138710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:40.138932 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.138879    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:20:40.138932 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.138902    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:20:40.138932 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.138914    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pd4dk for pod openshift-network-diagnostics/network-check-target-llj8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:40.139097 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.138974    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk podName:f64e39f7-de8e-43ea-987a-02c4811d7050 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:42.138954039 +0000 UTC m=+147.155786004 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pd4dk" (UniqueName: "kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk") pod "network-check-target-llj8p" (UID: "f64e39f7-de8e-43ea-987a-02c4811d7050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:40.373238 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.373101    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:20:40.445671 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:40.445607    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:40.445863 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.445731    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:40.446161 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:40.446117    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:40.446307 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:40.446239    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:40.505352 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:40.505303    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 13:15:38 +0000 UTC" deadline="2028-05-07 14:06:26.319957517 +0000 UTC"
Sep 10 13:20:40.505352 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:40.505352    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14520h45m45.814611627s"
Sep 10 13:20:41.448205 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:41.448105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:41.448665 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:41.448244    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:42.056629 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:42.056427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:42.056629 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:42.056477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:42.056629 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.056603    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:42.056909 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.056718    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs podName:25ca8aaa-ce03-426c-bc78-66e0fbebc64c nodeName:}" failed. No retries permitted until 2026-09-10 13:20:46.056696102 +0000 UTC m=+151.073528064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs") pod "network-metrics-daemon-6b7mv" (UID: "25ca8aaa-ce03-426c-bc78-66e0fbebc64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:42.056909 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.056609    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:42.056909 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.056828    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret podName:c9e7452e-8ac7-40b2-a170-fb36f5f4cff9 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:46.056808552 +0000 UTC m=+151.073640506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret") pod "global-pull-secret-syncer-62tzd" (UID: "c9e7452e-8ac7-40b2-a170-fb36f5f4cff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:42.158289 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:42.157629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:42.158289 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.157842    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:20:42.158289 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.157865    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:20:42.158289 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.157877    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pd4dk for pod openshift-network-diagnostics/network-check-target-llj8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:42.158289 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.157935    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk podName:f64e39f7-de8e-43ea-987a-02c4811d7050 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:46.15791737 +0000 UTC m=+151.174749335 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pd4dk" (UniqueName: "kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk") pod "network-check-target-llj8p" (UID: "f64e39f7-de8e-43ea-987a-02c4811d7050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:42.445290 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:42.445212    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:42.445441 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.445341    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:42.445732 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:42.445714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:42.445888 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:42.445820    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:43.445571 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:43.445539    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:43.445987 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:43.445679    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:44.204154 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:44.204122    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:20:44.445261 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:44.445052    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:44.445261 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:44.445063    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:44.445261 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:44.445213    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:44.445522 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:44.445265    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:45.374471 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:45.374416    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:20:45.445596 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:45.445566    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:45.446666 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:45.446630    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:46.087207 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:46.087152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:46.087380 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:46.087218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:46.087380 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.087323    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:46.087510 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.087380    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:46.087510 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.087402    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs podName:25ca8aaa-ce03-426c-bc78-66e0fbebc64c nodeName:}" failed. No retries permitted until 2026-09-10 13:20:54.087379726 +0000 UTC m=+159.104211695 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs") pod "network-metrics-daemon-6b7mv" (UID: "25ca8aaa-ce03-426c-bc78-66e0fbebc64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:46.087510 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.087430    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret podName:c9e7452e-8ac7-40b2-a170-fb36f5f4cff9 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:54.087416069 +0000 UTC m=+159.104248024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret") pod "global-pull-secret-syncer-62tzd" (UID: "c9e7452e-8ac7-40b2-a170-fb36f5f4cff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:46.187853 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:46.187815    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:46.188025 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.187994    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:20:46.188025 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.188019    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:20:46.188145 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.188029    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pd4dk for pod openshift-network-diagnostics/network-check-target-llj8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:46.188145 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.188088    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk podName:f64e39f7-de8e-43ea-987a-02c4811d7050 nodeName:}" failed. No retries permitted until 2026-09-10 13:20:54.188068272 +0000 UTC m=+159.204900223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pd4dk" (UniqueName: "kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk") pod "network-check-target-llj8p" (UID: "f64e39f7-de8e-43ea-987a-02c4811d7050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:46.445608 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:46.445521    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:46.445608 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:46.445570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:46.446086 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.445656    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:46.446086 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:46.445792    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:47.448487 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:47.448460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:47.448882 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:47.448581    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:48.446155 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:48.446079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:48.446330 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:48.446084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:48.446330 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:48.446212    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:48.446330 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:48.446286    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:49.445884 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:49.445849    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:49.446294 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:49.445957    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:50.375588 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:50.375545    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:20:50.445962 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:50.445927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:50.445962 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:50.445957    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:50.446432 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:50.446049    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:50.446432 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:50.446181    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:51.335486 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:51.335448    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-z52mw_kube-system(8fbe2e10-4ae2-4914-9e54-fa46bdc66b94): CreateContainerConfigError: services have not yet been read at least once, cannot construct envvars" logger="UnhandledError"
Sep 10 13:20:51.335639 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:51.335501    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with CreateContainerConfigError: \"services have not yet been read at least once, cannot construct envvars\"" pod="kube-system/konnectivity-agent-z52mw" podUID="8fbe2e10-4ae2-4914-9e54-fa46bdc66b94"
Sep 10 13:20:51.446191 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:51.446150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:51.446636 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:51.446318    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:52.445790 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:52.445755    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:52.445790 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:52.445787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:52.446048 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:52.445876    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:52.446048 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:52.445958    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:52.446403 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:52.446378    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:20:53.445157 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:53.445124    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:53.445326 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:53.445261    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:54.146687 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:54.146651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:54.146687 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:54.146691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:54.147165 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.146803    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:54.147165 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.146860    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret podName:c9e7452e-8ac7-40b2-a170-fb36f5f4cff9 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:10.14684675 +0000 UTC m=+175.163678700 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret") pod "global-pull-secret-syncer-62tzd" (UID: "c9e7452e-8ac7-40b2-a170-fb36f5f4cff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:20:54.147165 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.146804    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:54.147165 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.146980    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs podName:25ca8aaa-ce03-426c-bc78-66e0fbebc64c nodeName:}" failed. No retries permitted until 2026-09-10 13:21:10.146958443 +0000 UTC m=+175.163790392 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs") pod "network-metrics-daemon-6b7mv" (UID: "25ca8aaa-ce03-426c-bc78-66e0fbebc64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:20:54.247250 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:54.247214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:54.247433 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.247320    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:20:54.247433 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.247337    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:20:54.247433 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.247345    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pd4dk for pod openshift-network-diagnostics/network-check-target-llj8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:54.247433 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.247402    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk podName:f64e39f7-de8e-43ea-987a-02c4811d7050 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:10.247384856 +0000 UTC m=+175.264216824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pd4dk" (UniqueName: "kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk") pod "network-check-target-llj8p" (UID: "f64e39f7-de8e-43ea-987a-02c4811d7050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:20:54.445760 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:54.445683    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:54.445937 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:54.445686    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:54.445937 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.445802    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:54.445937 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:54.445917    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:54.509680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:54.509642    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:20:55.375972 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:55.375936    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:20:55.445902 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:55.445876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:55.445998 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:55.445957    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:55.765838 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:55.765676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerStarted","Data":"6110c70a99725c1e23433940b58c0ea2e522893318c5d3a955e08c1ee0a997ad"}
Sep 10 13:20:55.767557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:55.767529    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z52mw" event={"ID":"8fbe2e10-4ae2-4914-9e54-fa46bdc66b94","Type":"ContainerStarted","Data":"6ebfda7bad583ec42b57bf4188fdfa4ede3c99cfb45edad1b8348e3fbfa5a404"}
Sep 10 13:20:55.769827 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:55.769783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerStarted","Data":"ee95ac2d09c304390c897b0e2cd1ed25d486e099b614632641080199b0e405c3"}
Sep 10 13:20:55.771427 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:55.771400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8b7fp" event={"ID":"7cca75e3-1d89-4758-9cd2-ad515da5ffe4","Type":"ContainerStarted","Data":"32d69331be72bafc8b6691405e33d7945eae69319bcc151eba87eb9c837656f7"}
Sep 10 13:20:55.773121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:55.772949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-64j9z" event={"ID":"28616fa8-74fb-434f-8238-4246116c85b2","Type":"ContainerStarted","Data":"01e104786cae102358d528feedc2e149957e77920e645f4839e79a1a3a84a58e"}
Sep 10 13:20:55.775735 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:55.775355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h" event={"ID":"68b95818-d562-46cd-808a-c7a2df57ecdf","Type":"ContainerStarted","Data":"d48de0d33e390716464211b95b16fe3b1cbce17434cf4eca73a6b977696ebbb2"}
Sep 10 13:20:55.838896 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:55.838683    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z52mw" podStartSLOduration=15.21740418 podStartE2EDuration="27.838662046s" totalImagesPullingTime="12.621257866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:20:55.796365755 +0000 UTC m=+160.813197729" watchObservedRunningTime="2026-09-10 13:20:55.838662046 +0000 UTC m=+160.855494017"
Sep 10 13:20:55.859526 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:55.859466    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-64j9z" podStartSLOduration=12.061439052 podStartE2EDuration="28.859451522s" totalImagesPullingTime="16.79801247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:20:55.859373581 +0000 UTC m=+160.876205551" watchObservedRunningTime="2026-09-10 13:20:55.859451522 +0000 UTC m=+160.876283493"
Sep 10 13:20:56.445222 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.445123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:56.445222 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.445148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:56.445985 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:56.445298    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:56.445985 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:56.445424    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:56.779067 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.779041    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 13:20:56.779337 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.779313    2578 generic.go:356] "Generic (PLEG): container finished" podID="104356b6-35e4-472f-b715-aca908204fb4" containerID="ee95ac2d09c304390c897b0e2cd1ed25d486e099b614632641080199b0e405c3" exitCode=0
Sep 10 13:20:56.779426 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.779390    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerDied","Data":"ee95ac2d09c304390c897b0e2cd1ed25d486e099b614632641080199b0e405c3"}
Sep 10 13:20:56.780864 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.780840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s69g5" event={"ID":"5bf7fbef-791b-4e8c-8c6d-baed2bc964ee","Type":"ContainerStarted","Data":"7d3fcbd0f466ab48af8c4ba741ebcfb72c916e3df0ceef6428fb9c4b027ff74c"}
Sep 10 13:20:56.782622 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.782605    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h" event={"ID":"68b95818-d562-46cd-808a-c7a2df57ecdf","Type":"ContainerStarted","Data":"efeba27c7c14b28217dfc5500fc12295ab5de78f3fcc76085d1b53f67ba02851"}
Sep 10 13:20:56.783883 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.783862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xq466" event={"ID":"7e2ddbb0-f6a4-4610-8df2-55ff53f41368","Type":"ContainerStarted","Data":"3b0150e1c2e7dec302f152a8a34d1ccd8cf422772a2fad07c69202c6d3bafc9b"}
Sep 10 13:20:56.786709 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.786691    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerStarted","Data":"cc9e25e03b3acf344a04ee189b365df80d880b931c5fa8cd91f521d68b313ca4"}
Sep 10 13:20:56.786802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.786746    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerStarted","Data":"8f6c108a8b5edd917f2d17d79e322e27f757a5930c109c7bfa846f7ca4aa58e4"}
Sep 10 13:20:56.786802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.786777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerStarted","Data":"fb573ad4781cfb98bba8433ee6e360fe168114e72960727aa8479572dd249539"}
Sep 10 13:20:56.786900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.786809    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerStarted","Data":"572d6d9de4af11c14805ac81427f1c550251369dcfc99db9de6b28540a512712"}
Sep 10 13:20:56.786900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.786842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerStarted","Data":"ae96eda1c32c82e3103e0ce9b97953f530cf6e556d2b140c9b5a8cadef0030dc"}
Sep 10 13:20:56.802651 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.802612    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8b7fp" podStartSLOduration=12.012167795 podStartE2EDuration="28.80260108s" totalImagesPullingTime="16.790433285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:20:55.880818718 +0000 UTC m=+160.897650689" watchObservedRunningTime="2026-09-10 13:20:56.80260108 +0000 UTC m=+161.819433051"
Sep 10 13:20:56.849343 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.849294    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s69g5" podStartSLOduration=13.015165581 podStartE2EDuration="29.849277439s" totalImagesPullingTime="16.834111858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:20:56.833330269 +0000 UTC m=+161.850162238" watchObservedRunningTime="2026-09-10 13:20:56.849277439 +0000 UTC m=+161.866109436"
Sep 10 13:20:56.849588 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:56.849570    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xq466" podStartSLOduration=12.067466739 podStartE2EDuration="28.849564236s" totalImagesPullingTime="16.782097497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:20:56.849410746 +0000 UTC m=+161.866242699" watchObservedRunningTime="2026-09-10 13:20:56.849564236 +0000 UTC m=+161.866396207"
Sep 10 13:20:57.445325 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:57.445293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:57.445931 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:57.445453    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:57.453491 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:57.453397    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T13:20:56.779063582Z","UUID":"2f16d38b-3897-4cc3-b233-56150cd2bf1d","Handler":null,"Name":"","Endpoint":""}
Sep 10 13:20:57.454970 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:57.454947    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 13:20:57.455085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:57.454984    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 13:20:58.445746 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.445711    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:20:58.446149 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:58.445829    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:20:58.446149 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.445863    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:20:58.446149 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:58.445965    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:20:58.703337 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.703254    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:58.704278 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.704260    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:58.782870 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.782839    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-vjgdt"]
Sep 10 13:20:58.784559 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.784538    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:58.788036 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.788013    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6m2vk\""
Sep 10 13:20:58.788157 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.788129    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 13:20:58.788287 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.788261    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 13:20:58.792686 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.792635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h" event={"ID":"68b95818-d562-46cd-808a-c7a2df57ecdf","Type":"ContainerStarted","Data":"1aae4099f5dd65ec6033d2c0c2e04d2129f9bc377ac072da5d0b7f30b818f55e"}
Sep 10 13:20:58.796098 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.796076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerStarted","Data":"868a7e497645addcf3242388da3d73d72d52ecdc91d430180dea346b36e19560"}
Sep 10 13:20:58.796255 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.796238    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:58.796708 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.796693    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z52mw"
Sep 10 13:20:58.879262 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.879233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e22079e-1953-486a-8fca-835b0eb1092f-tmp-dir\") pod \"node-resolver-vjgdt\" (UID: \"2e22079e-1953-486a-8fca-835b0eb1092f\") " pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:58.879448 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.879342    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e22079e-1953-486a-8fca-835b0eb1092f-hosts-file\") pod \"node-resolver-vjgdt\" (UID: \"2e22079e-1953-486a-8fca-835b0eb1092f\") " pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:58.879448 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.879383    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ltcj\" (UniqueName: \"kubernetes.io/projected/2e22079e-1953-486a-8fca-835b0eb1092f-kube-api-access-5ltcj\") pod \"node-resolver-vjgdt\" (UID: \"2e22079e-1953-486a-8fca-835b0eb1092f\") " pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:58.980701 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.980522    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e22079e-1953-486a-8fca-835b0eb1092f-hosts-file\") pod \"node-resolver-vjgdt\" (UID: \"2e22079e-1953-486a-8fca-835b0eb1092f\") " pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:58.980855 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.980726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ltcj\" (UniqueName: \"kubernetes.io/projected/2e22079e-1953-486a-8fca-835b0eb1092f-kube-api-access-5ltcj\") pod \"node-resolver-vjgdt\" (UID: \"2e22079e-1953-486a-8fca-835b0eb1092f\") " pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:58.980855 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.980749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e22079e-1953-486a-8fca-835b0eb1092f-tmp-dir\") pod \"node-resolver-vjgdt\" (UID: \"2e22079e-1953-486a-8fca-835b0eb1092f\") " pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:58.980855 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.980665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2e22079e-1953-486a-8fca-835b0eb1092f-hosts-file\") pod \"node-resolver-vjgdt\" (UID: \"2e22079e-1953-486a-8fca-835b0eb1092f\") " pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:58.981230 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.981217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e22079e-1953-486a-8fca-835b0eb1092f-tmp-dir\") pod \"node-resolver-vjgdt\" (UID: \"2e22079e-1953-486a-8fca-835b0eb1092f\") " pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:58.990635 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:58.990604    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ltcj\" (UniqueName: \"kubernetes.io/projected/2e22079e-1953-486a-8fca-835b0eb1092f-kube-api-access-5ltcj\") pod \"node-resolver-vjgdt\" (UID: \"2e22079e-1953-486a-8fca-835b0eb1092f\") " pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:59.095226 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:59.095181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vjgdt"
Sep 10 13:20:59.104481 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:20:59.104442    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e22079e_1953_486a_8fca_835b0eb1092f.slice/crio-e4c744ac6523007b24c0c74f9d5b892ce84e7e2f890fcd91dcdf3e63e650b103 WatchSource:0}: Error finding container e4c744ac6523007b24c0c74f9d5b892ce84e7e2f890fcd91dcdf3e63e650b103: Status 404 returned error can't find the container with id e4c744ac6523007b24c0c74f9d5b892ce84e7e2f890fcd91dcdf3e63e650b103
Sep 10 13:20:59.445472 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:59.445384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:20:59.445635 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:20:59.445509    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:20:59.799463 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:59.799428    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjgdt" event={"ID":"2e22079e-1953-486a-8fca-835b0eb1092f","Type":"ContainerStarted","Data":"06653f66072d3145ba0f671640663a7a49f10f71d948aeeba54546935085b967"}
Sep 10 13:20:59.800183 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:59.799501    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vjgdt" event={"ID":"2e22079e-1953-486a-8fca-835b0eb1092f","Type":"ContainerStarted","Data":"e4c744ac6523007b24c0c74f9d5b892ce84e7e2f890fcd91dcdf3e63e650b103"}
Sep 10 13:20:59.820106 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:59.820057    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vjgdt" podStartSLOduration=1.820041077 podStartE2EDuration="1.820041077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:20:59.819447953 +0000 UTC m=+164.836279923" watchObservedRunningTime="2026-09-10 13:20:59.820041077 +0000 UTC m=+164.836873040"
Sep 10 13:20:59.820270 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:20:59.820238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j4n6h" podStartSLOduration=12.572639934 podStartE2EDuration="31.820231567s" totalImagesPullingTime="19.247591633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:20:58.835003452 +0000 UTC m=+163.851835422" watchObservedRunningTime="2026-09-10 13:20:59.820231567 +0000 UTC m=+164.837063590"
Sep 10 13:21:00.377086 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:00.377049    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:21:00.445736 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:00.445648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:00.445894 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:00.445773    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:00.445894 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:00.445823    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:00.446007 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:00.445944    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:01.445939 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.445777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:01.446458 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:01.446023    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:01.806107 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.806065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerStarted","Data":"810899c6912716975b515bafb065cba49e9375aeb4ff95f47f60d5cb077764ce"}
Sep 10 13:21:01.809663 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.809632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerStarted","Data":"71dd60db23c8b16a2ec598579d58d0eb54ed22d0801c0cdc6af6f8c35a2a174a"}
Sep 10 13:21:01.809916 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.809846    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="sbdb" containerID="cri-o://868a7e497645addcf3242388da3d73d72d52ecdc91d430180dea346b36e19560" gracePeriod=30
Sep 10 13:21:01.809916 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.809895    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="nbdb" containerID="cri-o://cc9e25e03b3acf344a04ee189b365df80d880b931c5fa8cd91f521d68b313ca4" gracePeriod=30
Sep 10 13:21:01.810055 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.809914    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovn-acl-logging" containerID="cri-o://ae96eda1c32c82e3103e0ce9b97953f530cf6e556d2b140c9b5a8cadef0030dc" gracePeriod=30
Sep 10 13:21:01.810055 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.809856    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="kube-rbac-proxy-ovn-metrics" containerID="cri-o://fb573ad4781cfb98bba8433ee6e360fe168114e72960727aa8479572dd249539" gracePeriod=30
Sep 10 13:21:01.810055 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.809956    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:21:01.810055 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.809845    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovn-controller" containerID="cri-o://6110c70a99725c1e23433940b58c0ea2e522893318c5d3a955e08c1ee0a997ad" gracePeriod=30
Sep 10 13:21:01.810055 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.809899    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="kube-rbac-proxy-node" containerID="cri-o://572d6d9de4af11c14805ac81427f1c550251369dcfc99db9de6b28540a512712" gracePeriod=30
Sep 10 13:21:01.810055 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.809940    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="northd" containerID="cri-o://8f6c108a8b5edd917f2d17d79e322e27f757a5930c109c7bfa846f7ca4aa58e4" gracePeriod=30
Sep 10 13:21:01.818888 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:01.818726    2578 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="868a7e497645addcf3242388da3d73d72d52ecdc91d430180dea346b36e19560" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:21:01.820540 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:01.820457    2578 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="868a7e497645addcf3242388da3d73d72d52ecdc91d430180dea346b36e19560" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:21:01.822234 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:01.821841    2578 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="868a7e497645addcf3242388da3d73d72d52ecdc91d430180dea346b36e19560" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Sep 10 13:21:01.822234 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:01.821888    2578 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="sbdb" probeResult="unknown"
Sep 10 13:21:01.832526 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.832493    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovnkube-controller" containerID="cri-o://71dd60db23c8b16a2ec598579d58d0eb54ed22d0801c0cdc6af6f8c35a2a174a" gracePeriod=30
Sep 10 13:21:01.858770 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:01.858723    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" podStartSLOduration=17.084142131 podStartE2EDuration="33.858704406s" totalImagesPullingTime="16.774562275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:01.857440235 +0000 UTC m=+166.874272206" watchObservedRunningTime="2026-09-10 13:21:01.858704406 +0000 UTC m=+166.875536377"
Sep 10 13:21:02.445410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.445372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:02.445544 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.445372    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:02.445544 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:02.445502    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:02.445604 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:02.445554    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:02.813064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.813037    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/ovnkube-controller/0.log"
Sep 10 13:21:02.814451 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.814434    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 13:21:02.814863 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.814849    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/kube-rbac-proxy-node/0.log"
Sep 10 13:21:02.815233 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815221    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/ovn-acl-logging/0.log"
Sep 10 13:21:02.815639 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815625    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/ovn-controller/0.log"
Sep 10 13:21:02.815691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815658    2578 generic.go:356] "Generic (PLEG): container finished" podID="60c25acb-86d9-426a-9685-1818f7c32000" containerID="71dd60db23c8b16a2ec598579d58d0eb54ed22d0801c0cdc6af6f8c35a2a174a" exitCode=1
Sep 10 13:21:02.815691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815673    2578 generic.go:356] "Generic (PLEG): container finished" podID="60c25acb-86d9-426a-9685-1818f7c32000" containerID="868a7e497645addcf3242388da3d73d72d52ecdc91d430180dea346b36e19560" exitCode=0
Sep 10 13:21:02.815691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815679    2578 generic.go:356] "Generic (PLEG): container finished" podID="60c25acb-86d9-426a-9685-1818f7c32000" containerID="cc9e25e03b3acf344a04ee189b365df80d880b931c5fa8cd91f521d68b313ca4" exitCode=0
Sep 10 13:21:02.815691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815685    2578 generic.go:356] "Generic (PLEG): container finished" podID="60c25acb-86d9-426a-9685-1818f7c32000" containerID="8f6c108a8b5edd917f2d17d79e322e27f757a5930c109c7bfa846f7ca4aa58e4" exitCode=0
Sep 10 13:21:02.815691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815689    2578 generic.go:356] "Generic (PLEG): container finished" podID="60c25acb-86d9-426a-9685-1818f7c32000" containerID="fb573ad4781cfb98bba8433ee6e360fe168114e72960727aa8479572dd249539" exitCode=143
Sep 10 13:21:02.815839 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815695    2578 generic.go:356] "Generic (PLEG): container finished" podID="60c25acb-86d9-426a-9685-1818f7c32000" containerID="572d6d9de4af11c14805ac81427f1c550251369dcfc99db9de6b28540a512712" exitCode=143
Sep 10 13:21:02.815839 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815701    2578 generic.go:356] "Generic (PLEG): container finished" podID="60c25acb-86d9-426a-9685-1818f7c32000" containerID="ae96eda1c32c82e3103e0ce9b97953f530cf6e556d2b140c9b5a8cadef0030dc" exitCode=143
Sep 10 13:21:02.815839 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815706    2578 generic.go:356] "Generic (PLEG): container finished" podID="60c25acb-86d9-426a-9685-1818f7c32000" containerID="6110c70a99725c1e23433940b58c0ea2e522893318c5d3a955e08c1ee0a997ad" exitCode=143
Sep 10 13:21:02.815839 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerDied","Data":"71dd60db23c8b16a2ec598579d58d0eb54ed22d0801c0cdc6af6f8c35a2a174a"}
Sep 10 13:21:02.815839 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerDied","Data":"868a7e497645addcf3242388da3d73d72d52ecdc91d430180dea346b36e19560"}
Sep 10 13:21:02.815839 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerDied","Data":"cc9e25e03b3acf344a04ee189b365df80d880b931c5fa8cd91f521d68b313ca4"}
Sep 10 13:21:02.816023 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerDied","Data":"8f6c108a8b5edd917f2d17d79e322e27f757a5930c109c7bfa846f7ca4aa58e4"}
Sep 10 13:21:02.816023 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerDied","Data":"fb573ad4781cfb98bba8433ee6e360fe168114e72960727aa8479572dd249539"}
Sep 10 13:21:02.816023 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerDied","Data":"572d6d9de4af11c14805ac81427f1c550251369dcfc99db9de6b28540a512712"}
Sep 10 13:21:02.816023 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815953    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerDied","Data":"ae96eda1c32c82e3103e0ce9b97953f530cf6e556d2b140c9b5a8cadef0030dc"}
Sep 10 13:21:02.816023 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.815981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerDied","Data":"6110c70a99725c1e23433940b58c0ea2e522893318c5d3a955e08c1ee0a997ad"}
Sep 10 13:21:02.816023 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.816023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m" event={"ID":"60c25acb-86d9-426a-9685-1818f7c32000","Type":"ContainerDied","Data":"311cc1b7a6735685758cf5563d9dc7d236b7530cd9c241205da18820ddb920c0"}
Sep 10 13:21:02.816212 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.816054    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="311cc1b7a6735685758cf5563d9dc7d236b7530cd9c241205da18820ddb920c0"
Sep 10 13:21:02.817572 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.817553    2578 generic.go:356] "Generic (PLEG): container finished" podID="104356b6-35e4-472f-b715-aca908204fb4" containerID="810899c6912716975b515bafb065cba49e9375aeb4ff95f47f60d5cb077764ce" exitCode=0
Sep 10 13:21:02.817645 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.817580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerDied","Data":"810899c6912716975b515bafb065cba49e9375aeb4ff95f47f60d5cb077764ce"}
Sep 10 13:21:02.824127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.824110    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/ovnkube-controller/0.log"
Sep 10 13:21:02.825468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.825451    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/kube-rbac-proxy-ovn-metrics/0.log"
Sep 10 13:21:02.825847 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.825832    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/kube-rbac-proxy-node/0.log"
Sep 10 13:21:02.826190 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.826176    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/ovn-acl-logging/0.log"
Sep 10 13:21:02.826574 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.826558    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6pv7m_60c25acb-86d9-426a-9685-1818f7c32000/ovn-controller/0.log"
Sep 10 13:21:02.845937 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.845913    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:21:02.928630 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928597    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-v5c7h"]
Sep 10 13:21:02.928783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928737    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="nbdb"
Sep 10 13:21:02.928783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928747    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="nbdb"
Sep 10 13:21:02.928783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928753    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="sbdb"
Sep 10 13:21:02.928783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928759    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="sbdb"
Sep 10 13:21:02.928783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928767    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovn-controller"
Sep 10 13:21:02.928783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928772    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovn-controller"
Sep 10 13:21:02.928783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928777    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="northd"
Sep 10 13:21:02.928783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928782    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="northd"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928789    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovnkube-controller"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928794    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovnkube-controller"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928799    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="kube-rbac-proxy-node"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928805    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="kube-rbac-proxy-node"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928810    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="kube-rbac-proxy-ovn-metrics"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928815    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="kube-rbac-proxy-ovn-metrics"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928822    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovn-acl-logging"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928827    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovn-acl-logging"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928855    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovnkube-controller"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928861    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="kube-rbac-proxy-ovn-metrics"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928866    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovn-acl-logging"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928872    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="kube-rbac-proxy-node"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928876    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="northd"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928881    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="nbdb"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928885    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="sbdb"
Sep 10 13:21:02.928993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.928888    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60c25acb-86d9-426a-9685-1818f7c32000" containerName="ovn-controller"
Sep 10 13:21:02.941795 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:02.941771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.008277 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-bin\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008277 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60c25acb-86d9-426a-9685-1818f7c32000-ovn-node-metrics-cert\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-systemd-units\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-env-overrides\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008348    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-etc-openvswitch\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-bin" (OuterVolumeSpecName: "host-cni-bin") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "host-cni-bin". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008368    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-kubelet\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008391    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-log-socket\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-systemd-units" (OuterVolumeSpecName: "systemd-units") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "systemd-units". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008420    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-var-lib-cni-networks-ovn-kubernetes\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008388    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-etc-openvswitch" (OuterVolumeSpecName: "etc-openvswitch") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "etc-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008404    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-kubelet" (OuterVolumeSpecName: "host-kubelet") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "host-kubelet". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008410    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-log-socket" (OuterVolumeSpecName: "log-socket") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "log-socket". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008444    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-systemd\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008460    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-var-lib-cni-networks-ovn-kubernetes" (OuterVolumeSpecName: "host-var-lib-cni-networks-ovn-kubernetes") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "host-var-lib-cni-networks-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008467    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-ovn\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008489    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-ovn" (OuterVolumeSpecName: "run-ovn") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "run-ovn". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.008515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008510    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-netns\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008530    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-netd\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-openvswitch\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008558    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-var-lib-openvswitch\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008582    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-netd" (OuterVolumeSpecName: "host-cni-netd") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "host-cni-netd". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lnhh\" (UniqueName: \"kubernetes.io/projected/60c25acb-86d9-426a-9685-1818f7c32000-kube-api-access-7lnhh\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008603    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-netns" (OuterVolumeSpecName: "host-run-netns") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "host-run-netns". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008607    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-var-lib-openvswitch" (OuterVolumeSpecName: "var-lib-openvswitch") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "var-lib-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008615    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-slash\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-node-log\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008701    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-script-lib\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008618    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-openvswitch" (OuterVolumeSpecName: "run-openvswitch") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "run-openvswitch". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008642    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-slash" (OuterVolumeSpecName: "host-slash") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "host-slash". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008727    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-config\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008739    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-node-log" (OuterVolumeSpecName: "node-log") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "node-log". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-ovn-kubernetes\") pod \"60c25acb-86d9-426a-9685-1818f7c32000\" (UID: \"60c25acb-86d9-426a-9685-1818f7c32000\") "
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008923    2578 reconciler_common.go:299] "Volume detached for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-netd\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-env-overrides" (OuterVolumeSpecName: "env-overrides") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "env-overrides". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008945    2578 reconciler_common.go:299] "Volume detached for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-openvswitch\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008960    2578 reconciler_common.go:299] "Volume detached for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-var-lib-openvswitch\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008973    2578 reconciler_common.go:299] "Volume detached for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-slash\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008984    2578 reconciler_common.go:299] "Volume detached for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-node-log\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008981    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-ovn-kubernetes" (OuterVolumeSpecName: "host-run-ovn-kubernetes") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "host-run-ovn-kubernetes". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.008995    2578 reconciler_common.go:299] "Volume detached for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-cni-bin\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.009007    2578 reconciler_common.go:299] "Volume detached for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-systemd-units\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.009018    2578 reconciler_common.go:299] "Volume detached for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-etc-openvswitch\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.009032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-script-lib" (OuterVolumeSpecName: "ovnkube-script-lib") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "ovnkube-script-lib". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.009044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-config" (OuterVolumeSpecName: "ovnkube-config") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "ovnkube-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.009027    2578 reconciler_common.go:299] "Volume detached for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-kubelet\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.009077    2578 reconciler_common.go:299] "Volume detached for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-log-socket\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.009090    2578 reconciler_common.go:299] "Volume detached for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-var-lib-cni-networks-ovn-kubernetes\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.009103    2578 reconciler_common.go:299] "Volume detached for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-ovn\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.009680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.009114    2578 reconciler_common.go:299] "Volume detached for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-netns\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.011464 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.011442    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c25acb-86d9-426a-9685-1818f7c32000-ovn-node-metrics-cert" (OuterVolumeSpecName: "ovn-node-metrics-cert") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "ovn-node-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:21:03.011623 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.011599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c25acb-86d9-426a-9685-1818f7c32000-kube-api-access-7lnhh" (OuterVolumeSpecName: "kube-api-access-7lnhh") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "kube-api-access-7lnhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:21:03.012278 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.012259    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-systemd" (OuterVolumeSpecName: "run-systemd") pod "60c25acb-86d9-426a-9685-1818f7c32000" (UID: "60c25acb-86d9-426a-9685-1818f7c32000"). InnerVolumeSpecName "run-systemd". PluginName "kubernetes.io/host-path", VolumeGIDValue ""
Sep 10 13:21:03.109571 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-run-ovn-kubernetes\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.109571 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-systemd-units\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.109782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-node-log\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.109782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-cni-bin\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.109782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109681    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-run-ovn\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.109782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109700    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-run-systemd\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.109782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.109782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-ovn-node-metrics-cert\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109792    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-ovnkube-script-lib\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-run-netns\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-etc-openvswitch\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-slash\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-log-socket\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-kubelet\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.109998    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-env-overrides\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mk74\" (UniqueName: \"kubernetes.io/projected/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-kube-api-access-5mk74\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-ovnkube-config\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-var-lib-openvswitch\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110117    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-run-openvswitch\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-cni-netd\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110167    2578 reconciler_common.go:299] "Volume detached for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-host-run-ovn-kubernetes\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110215    2578 reconciler_common.go:299] "Volume detached for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/60c25acb-86d9-426a-9685-1818f7c32000-ovn-node-metrics-cert\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110226    2578 reconciler_common.go:299] "Volume detached for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-env-overrides\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110235    2578 reconciler_common.go:299] "Volume detached for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/60c25acb-86d9-426a-9685-1818f7c32000-run-systemd\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110243    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lnhh\" (UniqueName: \"kubernetes.io/projected/60c25acb-86d9-426a-9685-1818f7c32000-kube-api-access-7lnhh\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110253    2578 reconciler_common.go:299] "Volume detached for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-script-lib\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.110310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.110262    2578 reconciler_common.go:299] "Volume detached for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/60c25acb-86d9-426a-9685-1818f7c32000-ovnkube-config\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:21:03.210843 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-run-ovn-kubernetes\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.210843 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-systemd-units\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-node-log\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-cni-bin\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-run-ovn\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-run-ovn-kubernetes\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-node-log\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-run-systemd\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210970    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-cni-bin\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-systemd-units\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-run-systemd\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.210994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-run-ovn\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-ovn-node-metrics-cert\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-ovnkube-script-lib\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-run-netns\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-etc-openvswitch\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-slash\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-log-socket\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-kubelet\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-env-overrides\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211236    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-etc-openvswitch\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mk74\" (UniqueName: \"kubernetes.io/projected/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-kube-api-access-5mk74\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211257    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-kubelet\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211260    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-run-netns\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-ovnkube-config\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-slash\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-var-lib-openvswitch\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211346    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-log-socket\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-run-openvswitch\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-cni-netd\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-var-lib-openvswitch\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-host-cni-netd\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.211681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-run-openvswitch\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.212129 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-env-overrides\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.212129 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-ovnkube-script-lib\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.212129 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.211763    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-ovnkube-config\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.213328 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.213312    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-ovn-node-metrics-cert\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.225099 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.225074    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mk74\" (UniqueName: \"kubernetes.io/projected/5bd1acf3-5d7c-4ee0-bc1e-53652ee68803-kube-api-access-5mk74\") pod \"ovnkube-node-v5c7h\" (UID: \"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803\") " pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.249303 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.249266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:03.288946 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:03.288922    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd1acf3_5d7c_4ee0_bc1e_53652ee68803.slice/crio-8dce7a88146d2cdb8df8cd12ec402e79e641d2f19769f5a69a0eb29fc8c21a29 WatchSource:0}: Error finding container 8dce7a88146d2cdb8df8cd12ec402e79e641d2f19769f5a69a0eb29fc8c21a29: Status 404 returned error can't find the container with id 8dce7a88146d2cdb8df8cd12ec402e79e641d2f19769f5a69a0eb29fc8c21a29
Sep 10 13:21:03.372806 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.372781    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 13:21:03.446092 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.446068    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:03.446234 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:03.446169    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:03.822336 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.822076    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" event={"ID":"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803","Type":"ContainerStarted","Data":"c52458b76fb2da75a0620d183db5c5f1c525ec977ef9227a93aeb6b76de03c5a"}
Sep 10 13:21:03.822605 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.822111    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6pv7m"
Sep 10 13:21:03.822605 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.822408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" event={"ID":"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803","Type":"ContainerStarted","Data":"2561169be4516d386b0bf927b50a58adf5159e4befddef201630b913dee149c8"}
Sep 10 13:21:03.822605 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.822525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" event={"ID":"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803","Type":"ContainerStarted","Data":"2ad86487dfc68a07f95e98fa3dafa426fafc26e8e5040fded1ec7aa4bff0bf06"}
Sep 10 13:21:03.822605 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.822553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" event={"ID":"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803","Type":"ContainerStarted","Data":"8dce7a88146d2cdb8df8cd12ec402e79e641d2f19769f5a69a0eb29fc8c21a29"}
Sep 10 13:21:03.854009 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.853983    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-6pv7m"]
Sep 10 13:21:03.861288 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:03.861268    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-6pv7m"]
Sep 10 13:21:04.445783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:04.445747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:04.445783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:04.445771    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:04.445948 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:04.445858    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:04.445992 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:04.445971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:04.827910 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:04.827881    2578 generic.go:356] "Generic (PLEG): container finished" podID="104356b6-35e4-472f-b715-aca908204fb4" containerID="ef0721a30dc488642747dd999ce581daa06b49a705cb9a45a81a2c94dbc50f81" exitCode=0
Sep 10 13:21:04.828369 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:04.827969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerDied","Data":"ef0721a30dc488642747dd999ce581daa06b49a705cb9a45a81a2c94dbc50f81"}
Sep 10 13:21:04.831019 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:04.830995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" event={"ID":"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803","Type":"ContainerStarted","Data":"c629e7b98741626f064d8a728ef41d2d0968012902c7b3ea854b5e069832d949"}
Sep 10 13:21:04.831087 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:04.831067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" event={"ID":"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803","Type":"ContainerStarted","Data":"db09dc3e4477802f517e0615238920d8f9bbb4bd1bde088ac5371df05ec06823"}
Sep 10 13:21:04.831127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:04.831108    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" event={"ID":"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803","Type":"ContainerStarted","Data":"53b185e6437296d652a6951b4a2f7e64b1f535adaffddbca826cf4376a07a105"}
Sep 10 13:21:05.377921 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:05.377887    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:21:05.446216 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:05.446172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:05.446385 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:05.446275    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:05.446720 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:05.446694    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:21:05.448718 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:05.448696    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60c25acb-86d9-426a-9685-1818f7c32000" path="/var/lib/kubelet/pods/60c25acb-86d9-426a-9685-1818f7c32000/volumes"
Sep 10 13:21:06.445055 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:06.445025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:06.445556 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:06.445038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:06.445556 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:06.445121    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:06.445556 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:06.445279    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:06.839065 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:06.839034    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" event={"ID":"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803","Type":"ContainerStarted","Data":"a1c4c875ceeab6795b7df9b3d1b6ad82392aaf1e6ede2bf1f122748165ae95fe"}
Sep 10 13:21:06.841204 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:06.841170    2578 generic.go:356] "Generic (PLEG): container finished" podID="104356b6-35e4-472f-b715-aca908204fb4" containerID="eb1fff1259bcfe9607e599f27b0f3a34c867d6782e3b1bfdc7efca9abdb74a20" exitCode=0
Sep 10 13:21:06.841309 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:06.841211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerDied","Data":"eb1fff1259bcfe9607e599f27b0f3a34c867d6782e3b1bfdc7efca9abdb74a20"}
Sep 10 13:21:07.445683 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:07.445648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:07.446114 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:07.445786    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:08.445659 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:08.445623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:08.446472 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:08.445744    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:08.446472 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:08.445805    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:08.446472 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:08.445929    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:08.851464 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:08.851429    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" event={"ID":"5bd1acf3-5d7c-4ee0-bc1e-53652ee68803","Type":"ContainerStarted","Data":"fc13c45f50bac6c2ea9f27efcf83c8036063c4abc48a5f05ef40e2ffcafbfb42"}
Sep 10 13:21:08.851746 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:08.851703    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:08.868098 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:08.868069    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:08.883090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:08.883041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h" podStartSLOduration=6.883022081 podStartE2EDuration="6.883022081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:08.881445384 +0000 UTC m=+173.898277354" watchObservedRunningTime="2026-09-10 13:21:08.883022081 +0000 UTC m=+173.899854053"
Sep 10 13:21:09.445340 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:09.445305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:09.445512 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:09.445417    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:09.855094 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:09.855032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:09.855889 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:09.855126    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:09.871869 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:09.871839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:10.158971 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:10.158879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:10.158971 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:10.158926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:10.159161 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.159033    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 13:21:10.159161 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.159093    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret podName:c9e7452e-8ac7-40b2-a170-fb36f5f4cff9 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:42.159075762 +0000 UTC m=+207.175907725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret") pod "global-pull-secret-syncer-62tzd" (UID: "c9e7452e-8ac7-40b2-a170-fb36f5f4cff9") : object "kube-system"/"original-pull-secret" not registered
Sep 10 13:21:10.159161 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.159034    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:21:10.159286 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.159173    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs podName:25ca8aaa-ce03-426c-bc78-66e0fbebc64c nodeName:}" failed. No retries permitted until 2026-09-10 13:21:42.159157508 +0000 UTC m=+207.175989473 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs") pod "network-metrics-daemon-6b7mv" (UID: "25ca8aaa-ce03-426c-bc78-66e0fbebc64c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 13:21:10.259323 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:10.259284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:10.259510 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.259464    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 13:21:10.259510 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.259500    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 13:21:10.259607 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.259514    2578 projected.go:196] Error preparing data for projected volume kube-api-access-pd4dk for pod openshift-network-diagnostics/network-check-target-llj8p: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:21:10.259607 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.259580    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk podName:f64e39f7-de8e-43ea-987a-02c4811d7050 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:42.259560813 +0000 UTC m=+207.276392776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pd4dk" (UniqueName: "kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk") pod "network-check-target-llj8p" (UID: "f64e39f7-de8e-43ea-987a-02c4811d7050") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 13:21:10.378985 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.378940    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:21:10.445508 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:10.445430    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:10.445508 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:10.445476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:10.445719 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.445577    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:10.445719 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:10.445684    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:11.446184 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:11.446153    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:11.446676 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:11.446284    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:11.865128 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:11.864844    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-62tzd"]
Sep 10 13:21:11.865550 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:11.865354    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:11.866444 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:11.866401    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:11.871992 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:11.871438    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-llj8p"]
Sep 10 13:21:11.871992 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:11.871591    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:11.871992 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:11.871790    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:11.889275 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:11.889246    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6b7mv"]
Sep 10 13:21:11.889432 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:11.889359    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:11.889474 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:11.889444    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:12.870375 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:12.870292    2578 generic.go:356] "Generic (PLEG): container finished" podID="104356b6-35e4-472f-b715-aca908204fb4" containerID="32d6add1f825ad404da6af78c7305f6ba78394ac71e0bfcf437a432ca5e292d8" exitCode=0
Sep 10 13:21:12.870375 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:12.870334    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerDied","Data":"32d6add1f825ad404da6af78c7305f6ba78394ac71e0bfcf437a432ca5e292d8"}
Sep 10 13:21:13.445425 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:13.445386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:13.445598 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:13.445390    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:13.445598 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:13.445509    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:13.445598 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:13.445514    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:13.445598 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:13.445590    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:13.445824 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:13.445658    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:13.875206 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:13.875160    2578 generic.go:356] "Generic (PLEG): container finished" podID="104356b6-35e4-472f-b715-aca908204fb4" containerID="3fd94b648f6835e595fa1baca49e3595169f17995508286071410b5d9e72ea9b" exitCode=0
Sep 10 13:21:13.875206 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:13.875209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerDied","Data":"3fd94b648f6835e595fa1baca49e3595169f17995508286071410b5d9e72ea9b"}
Sep 10 13:21:14.881674 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:14.881642    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r672x" event={"ID":"104356b6-35e4-472f-b715-aca908204fb4","Type":"ContainerStarted","Data":"2ae17663d2719abde5aa6f0c5f2ae46aa8ee5f5bb4b1167880d41d12410642c7"}
Sep 10 13:21:15.380348 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:15.380173    2578 kubelet.go:3032] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?"
Sep 10 13:21:15.446736 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:15.446705    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:15.446897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:15.446779    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:15.446897 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:15.446784    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:15.446897 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:15.446877    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:15.446995 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:15.446919    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:15.446995 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:15.446971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:16.446104 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:16.446016    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:21:17.445307 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:17.445272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:17.445456 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:17.445271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:17.445456 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:17.445387    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:17.445524 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:17.445280    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:17.445524 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:17.445465    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:17.445524 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:17.445521    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:19.435851 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:19.435824    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjgdt_2e22079e-1953-486a-8fca-835b0eb1092f/dns-node-resolver/0.log"
Sep 10 13:21:19.446041 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:19.446027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:19.446093 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:19.446046    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:19.446132 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:19.446027    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:19.446132 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:19.446119    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-llj8p" podUID="f64e39f7-de8e-43ea-987a-02c4811d7050"
Sep 10 13:21:19.446253 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:19.446232    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6b7mv" podUID="25ca8aaa-ce03-426c-bc78-66e0fbebc64c"
Sep 10 13:21:19.446333 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:19.446314    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-62tzd" podUID="c9e7452e-8ac7-40b2-a170-fb36f5f4cff9"
Sep 10 13:21:19.803310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:19.803287    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-64j9z_28616fa8-74fb-434f-8238-4246116c85b2/node-ca/0.log"
Sep 10 13:21:21.445449 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:21.445376    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:21.445909 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:21.445381    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:21.445909 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:21.445383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:21.450297 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:21.450267    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 13:21:21.450431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:21.450311    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 13:21:21.450431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:21.450322    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 13:21:21.450431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:21.450328    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-67hjc\""
Sep 10 13:21:21.450431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:21.450268    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 13:21:21.450431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:21.450323    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4sjdq\""
Sep 10 13:21:27.445698 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:27.445663    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:21:29.230399 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:29.230370    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-19.ec2.internal" event="NodeReady"
Sep 10 13:21:41.445806 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:41.445764    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:21:41.875882 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:41.875852    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v5c7h"
Sep 10 13:21:41.935613 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:41.935562    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r672x" podStartSLOduration=44.94729157 podStartE2EDuration="1m13.935546249s" totalImagesPullingTime="28.988254679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:14.9134092 +0000 UTC m=+179.930241170" watchObservedRunningTime="2026-09-10 13:21:41.935546249 +0000 UTC m=+206.952378219"
Sep 10 13:21:42.169421 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.169323    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:42.169421 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.169363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:42.173012 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.172985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25ca8aaa-ce03-426c-bc78-66e0fbebc64c-metrics-certs\") pod \"network-metrics-daemon-6b7mv\" (UID: \"25ca8aaa-ce03-426c-bc78-66e0fbebc64c\") " pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:42.173121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.173093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c9e7452e-8ac7-40b2-a170-fb36f5f4cff9-original-pull-secret\") pod \"global-pull-secret-syncer-62tzd\" (UID: \"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9\") " pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:42.270127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.270089    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:42.272481 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.272463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd4dk\" (UniqueName: \"kubernetes.io/projected/f64e39f7-de8e-43ea-987a-02c4811d7050-kube-api-access-pd4dk\") pod \"network-check-target-llj8p\" (UID: \"f64e39f7-de8e-43ea-987a-02c4811d7050\") " pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:42.455449 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.455355    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6b7mv"
Sep 10 13:21:42.460783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.460767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:42.465494 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.465471    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-62tzd"
Sep 10 13:21:42.758212 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.758164    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-62tzd"]
Sep 10 13:21:42.760325 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.760306    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6b7mv"]
Sep 10 13:21:42.762144 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:42.762114    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e7452e_8ac7_40b2_a170_fb36f5f4cff9.slice/crio-6382dd631d3d2d8bf95e2f5d9d6f90ce64940e35789361f7366b68ddbbc86486 WatchSource:0}: Error finding container 6382dd631d3d2d8bf95e2f5d9d6f90ce64940e35789361f7366b68ddbbc86486: Status 404 returned error can't find the container with id 6382dd631d3d2d8bf95e2f5d9d6f90ce64940e35789361f7366b68ddbbc86486
Sep 10 13:21:42.762449 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:42.762428    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ca8aaa_ce03_426c_bc78_66e0fbebc64c.slice/crio-e60a34698be143474795c87c9da0d18c475466eb09ccbaf74f9950d67659a9ea WatchSource:0}: Error finding container e60a34698be143474795c87c9da0d18c475466eb09ccbaf74f9950d67659a9ea: Status 404 returned error can't find the container with id e60a34698be143474795c87c9da0d18c475466eb09ccbaf74f9950d67659a9ea
Sep 10 13:21:42.779032 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.779005    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-llj8p"]
Sep 10 13:21:42.782048 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:42.782024    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64e39f7_de8e_43ea_987a_02c4811d7050.slice/crio-569c885a43f5cfaf7997d596ba871dd608d8059db0d11106c2cda69b68add471 WatchSource:0}: Error finding container 569c885a43f5cfaf7997d596ba871dd608d8059db0d11106c2cda69b68add471: Status 404 returned error can't find the container with id 569c885a43f5cfaf7997d596ba871dd608d8059db0d11106c2cda69b68add471
Sep 10 13:21:42.959286 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.959246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-62tzd" event={"ID":"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9","Type":"ContainerStarted","Data":"6382dd631d3d2d8bf95e2f5d9d6f90ce64940e35789361f7366b68ddbbc86486"}
Sep 10 13:21:42.960239 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.960219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-llj8p" event={"ID":"f64e39f7-de8e-43ea-987a-02c4811d7050","Type":"ContainerStarted","Data":"569c885a43f5cfaf7997d596ba871dd608d8059db0d11106c2cda69b68add471"}
Sep 10 13:21:42.961149 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:42.961133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6b7mv" event={"ID":"25ca8aaa-ce03-426c-bc78-66e0fbebc64c","Type":"ContainerStarted","Data":"e60a34698be143474795c87c9da0d18c475466eb09ccbaf74f9950d67659a9ea"}
Sep 10 13:21:44.972692 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:44.972644    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6b7mv" event={"ID":"25ca8aaa-ce03-426c-bc78-66e0fbebc64c","Type":"ContainerStarted","Data":"c0b48324f2d467bffba93334f517c0d6f2553b6139b108e4bebea109096cd822"}
Sep 10 13:21:44.973125 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:44.972743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6b7mv" event={"ID":"25ca8aaa-ce03-426c-bc78-66e0fbebc64c","Type":"ContainerStarted","Data":"ce51625b63ccddf26bcd6800479f889032117de5f43bf45ab5f19197294ce16e"}
Sep 10 13:21:45.001010 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.000951    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6b7mv" podStartSLOduration=76.42585 podStartE2EDuration="1m18.000928805s" totalImagesPullingTime="1.575078805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:20:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:44.999406198 +0000 UTC m=+210.016238170" watchObservedRunningTime="2026-09-10 13:21:45.000928805 +0000 UTC m=+210.017760777"
Sep 10 13:21:45.796029 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.795994    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"]
Sep 10 13:21:45.797893 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.797868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"
Sep 10 13:21:45.807954 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.807932    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 10 13:21:45.808090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.807934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qjcr4\""
Sep 10 13:21:45.812420 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.812349    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rnmmb"]
Sep 10 13:21:45.814371 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.814353    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-576868dd66-clmnw"]
Sep 10 13:21:45.814513 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.814493    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.816284 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.816267    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.816811 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.816781    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Sep 10 13:21:45.822433 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.822410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"]
Sep 10 13:21:45.825102 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.825034    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap"
Sep 10 13:21:45.825235 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.825110    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-pbl4r\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbl4r\"" type="*v1.Secret"
Sep 10 13:21:45.825380 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.825349    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 10 13:21:45.825507 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.825471    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 10 13:21:45.825617 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.825594    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"trusted-ca\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap"
Sep 10 13:21:45.825790 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.825758    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"registry-dockercfg-gf67b\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gf67b\"" type="*v1.Secret"
Sep 10 13:21:45.825866 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.825784    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"installation-pull-secrets\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret"
Sep 10 13:21:45.826021 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.825904    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-private-configuration\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret"
Sep 10 13:21:45.826021 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:45.825994    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"image-registry-tls\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-image-registry\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret"
Sep 10 13:21:45.851988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.851953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-576868dd66-clmnw"]
Sep 10 13:21:45.854075 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.854050    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rnmmb"]
Sep 10 13:21:45.889032 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.889003    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ch54g"]
Sep 10 13:21:45.891285 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:45.891430 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-image-registry-private-configuration\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.891430 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-installation-pull-secrets\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.891430 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891402    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d2dd98c8-d5d5-422a-a507-1480a9609412-crio-socket\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.891604 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2dd98c8-d5d5-422a-a507-1480a9609412-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.891604 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2dd98c8-d5d5-422a-a507-1480a9609412-data-volume\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.891604 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-trusted-ca\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.891604 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-registry-tls\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.891759 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-registry-certificates\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.891759 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f87cd7ce-82ce-4272-b867-47107fbaa79d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-85hmm\" (UID: \"f87cd7ce-82ce-4272-b867-47107fbaa79d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"
Sep 10 13:21:45.891847 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vs6\" (UniqueName: \"kubernetes.io/projected/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-api-access-r7vs6\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.891847 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wxk\" (UniqueName: \"kubernetes.io/projected/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-kube-api-access-b9wxk\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.891847 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.891847 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-ca-trust-extracted\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.892022 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.891877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-bound-sa-token\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.894499 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.894473    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 13:21:45.896796 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.896638    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7k2sl\""
Sep 10 13:21:45.896796 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.896680    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 13:21:45.951761 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.951732    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ch54g"]
Sep 10 13:21:45.992146 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d2dd98c8-d5d5-422a-a507-1480a9609412-crio-socket\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-config-volume\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2dd98c8-d5d5-422a-a507-1480a9609412-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2dd98c8-d5d5-422a-a507-1480a9609412-data-volume\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992240    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-trusted-ca\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-registry-tls\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d2dd98c8-d5d5-422a-a507-1480a9609412-crio-socket\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-registry-certificates\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f87cd7ce-82ce-4272-b867-47107fbaa79d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-85hmm\" (UID: \"f87cd7ce-82ce-4272-b867-47107fbaa79d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992378    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-metrics-tls\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992401    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-tmp-dir\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2l6b\" (UniqueName: \"kubernetes.io/projected/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-kube-api-access-n2l6b\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vs6\" (UniqueName: \"kubernetes.io/projected/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-api-access-r7vs6\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wxk\" (UniqueName: \"kubernetes.io/projected/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-kube-api-access-b9wxk\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-ca-trust-extracted\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.992610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-bound-sa-token\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.993213 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992584    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-image-registry-private-configuration\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.993213 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-installation-pull-secrets\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.993213 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992609    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2dd98c8-d5d5-422a-a507-1480a9609412-data-volume\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:45.993213 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.992924    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-ca-trust-extracted\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.993340 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.993264    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-registry-certificates\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:45.994614 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:45.994597    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f87cd7ce-82ce-4272-b867-47107fbaa79d-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-85hmm\" (UID: \"f87cd7ce-82ce-4272-b867-47107fbaa79d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"
Sep 10 13:21:46.012452 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.012424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-bound-sa-token\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:46.040650 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.040617    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wxk\" (UniqueName: \"kubernetes.io/projected/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-kube-api-access-b9wxk\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:46.054228 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.054139    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6976s"]
Sep 10 13:21:46.065347 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.065315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6976s"
Sep 10 13:21:46.068460 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.068428    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 13:21:46.068460 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.068445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 13:21:46.068807 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.068784    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 13:21:46.068925 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.068788    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nzsjz\""
Sep 10 13:21:46.093557 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.093528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-config-volume\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:46.093751 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.093571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-metrics-tls\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:46.093751 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.093590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-tmp-dir\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:46.093751 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.093612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2l6b\" (UniqueName: \"kubernetes.io/projected/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-kube-api-access-n2l6b\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:46.093751 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.093638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14d54b58-eea2-4440-b09f-6445e29204e4-cert\") pod \"ingress-canary-6976s\" (UID: \"14d54b58-eea2-4440-b09f-6445e29204e4\") " pod="openshift-ingress-canary/ingress-canary-6976s"
Sep 10 13:21:46.093751 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.093718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgsw\" (UniqueName: \"kubernetes.io/projected/14d54b58-eea2-4440-b09f-6445e29204e4-kube-api-access-ljgsw\") pod \"ingress-canary-6976s\" (UID: \"14d54b58-eea2-4440-b09f-6445e29204e4\") " pod="openshift-ingress-canary/ingress-canary-6976s"
Sep 10 13:21:46.094243 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.093945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-tmp-dir\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:46.094294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.094246    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-config-volume\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:46.096336 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.096311    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-metrics-tls\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:46.105573 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.105551    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6976s"]
Sep 10 13:21:46.106460 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.106437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"
Sep 10 13:21:46.119564 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.119540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2l6b\" (UniqueName: \"kubernetes.io/projected/63ac7b44-01ac-4815-8cb8-fa1af93a57f7-kube-api-access-n2l6b\") pod \"dns-default-ch54g\" (UID: \"63ac7b44-01ac-4815-8cb8-fa1af93a57f7\") " pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:46.194709 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.194666    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14d54b58-eea2-4440-b09f-6445e29204e4-cert\") pod \"ingress-canary-6976s\" (UID: \"14d54b58-eea2-4440-b09f-6445e29204e4\") " pod="openshift-ingress-canary/ingress-canary-6976s"
Sep 10 13:21:46.194891 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.194729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgsw\" (UniqueName: \"kubernetes.io/projected/14d54b58-eea2-4440-b09f-6445e29204e4-kube-api-access-ljgsw\") pod \"ingress-canary-6976s\" (UID: \"14d54b58-eea2-4440-b09f-6445e29204e4\") " pod="openshift-ingress-canary/ingress-canary-6976s"
Sep 10 13:21:46.197210 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.197179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/14d54b58-eea2-4440-b09f-6445e29204e4-cert\") pod \"ingress-canary-6976s\" (UID: \"14d54b58-eea2-4440-b09f-6445e29204e4\") " pod="openshift-ingress-canary/ingress-canary-6976s"
Sep 10 13:21:46.202220 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.202183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:46.219658 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.219635    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgsw\" (UniqueName: \"kubernetes.io/projected/14d54b58-eea2-4440-b09f-6445e29204e4-kube-api-access-ljgsw\") pod \"ingress-canary-6976s\" (UID: \"14d54b58-eea2-4440-b09f-6445e29204e4\") " pod="openshift-ingress-canary/ingress-canary-6976s"
Sep 10 13:21:46.376110 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.376034    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6976s"
Sep 10 13:21:46.758607 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.758560    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 13:21:46.765578 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.765546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-installation-pull-secrets\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:46.765736 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.765596    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pbl4r\""
Sep 10 13:21:46.780503 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.780468    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 13:21:46.785103 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:46.785070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-registry-tls\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:46.993174 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:46.993126    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:46.993174 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:46.993154    2578 secret.go:189] Couldn't get secret openshift-image-registry/image-registry-private-configuration: failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:46.993807 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:46.993124    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:21:46.993807 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:46.993236    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2dd98c8-d5d5-422a-a507-1480a9609412-insights-runtime-extractor-tls podName:d2dd98c8-d5d5-422a-a507-1480a9609412 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:47.493212934 +0000 UTC m=+212.510044889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d2dd98c8-d5d5-422a-a507-1480a9609412-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rnmmb" (UID: "d2dd98c8-d5d5-422a-a507-1480a9609412") : failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:46.993807 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:46.993257    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-image-registry-private-configuration podName:36f9a8c0-7e20-40de-b964-0ba36aab5ea0 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:47.493247001 +0000 UTC m=+212.510078955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "image-registry-private-configuration" (UniqueName: "kubernetes.io/secret/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-image-registry-private-configuration") pod "image-registry-576868dd66-clmnw" (UID: "36f9a8c0-7e20-40de-b964-0ba36aab5ea0") : failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:46.993807 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:46.993130    2578 configmap.go:193] Couldn't get configMap openshift-image-registry/trusted-ca: failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:21:46.993807 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:46.993302    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-rbac-proxy-cm podName:d2dd98c8-d5d5-422a-a507-1480a9609412 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:47.493283208 +0000 UTC m=+212.510115163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rnmmb" (UID: "d2dd98c8-d5d5-422a-a507-1480a9609412") : failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:21:46.993807 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:46.993351    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-trusted-ca podName:36f9a8c0-7e20-40de-b964-0ba36aab5ea0 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:47.493338705 +0000 UTC m=+212.510170668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca" (UniqueName: "kubernetes.io/configmap/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-trusted-ca") pod "image-registry-576868dd66-clmnw" (UID: "36f9a8c0-7e20-40de-b964-0ba36aab5ea0") : failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:21:47.011900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.011826    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 13:21:47.039173 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:47.039142    2578 projected.go:291] Couldn't get configMap openshift-insights/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:21:47.042166 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.042144    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gf67b\""
Sep 10 13:21:47.064047 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.064016    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 13:21:47.225209 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.225173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 13:21:47.240431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.240399    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 13:21:47.249813 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:47.249779    2578 projected.go:196] Error preparing data for projected volume kube-api-access-r7vs6 for pod openshift-insights/insights-runtime-extractor-rnmmb: failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:21:47.249979 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:47.249868    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-api-access-r7vs6 podName:d2dd98c8-d5d5-422a-a507-1480a9609412 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:47.749847403 +0000 UTC m=+212.766679365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r7vs6" (UniqueName: "kubernetes.io/projected/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-api-access-r7vs6") pod "insights-runtime-extractor-rnmmb" (UID: "d2dd98c8-d5d5-422a-a507-1480a9609412") : failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:21:47.386763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.386697    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 13:21:47.413904 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.413867    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 13:21:47.509231 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.509181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-trusted-ca\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:47.509428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.509277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:47.509428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.509310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-image-registry-private-configuration\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:47.509428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.509348    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2dd98c8-d5d5-422a-a507-1480a9609412-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:47.509993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.509731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:47.509993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.509876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-trusted-ca\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:47.511879 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.511838    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/36f9a8c0-7e20-40de-b964-0ba36aab5ea0-image-registry-private-configuration\") pod \"image-registry-576868dd66-clmnw\" (UID: \"36f9a8c0-7e20-40de-b964-0ba36aab5ea0\") " pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:47.512148 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.512123    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d2dd98c8-d5d5-422a-a507-1480a9609412-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:47.633010 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.632966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:47.696266 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.696235    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"]
Sep 10 13:21:47.705308 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:47.705275    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87cd7ce_82ce_4272_b867_47107fbaa79d.slice/crio-0f30ccf639ebaba40111040ad96b39e1a5809283d1cb0a890d01e6e435d7f658 WatchSource:0}: Error finding container 0f30ccf639ebaba40111040ad96b39e1a5809283d1cb0a890d01e6e435d7f658: Status 404 returned error can't find the container with id 0f30ccf639ebaba40111040ad96b39e1a5809283d1cb0a890d01e6e435d7f658
Sep 10 13:21:47.760366 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.760335    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6976s"]
Sep 10 13:21:47.761151 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.761132    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ch54g"]
Sep 10 13:21:47.762596 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:47.762573    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ac7b44_01ac_4815_8cb8_fa1af93a57f7.slice/crio-e882e1575c6535cef951ce4285244a5f299bc0b337af579c6b6f3833984ba04d WatchSource:0}: Error finding container e882e1575c6535cef951ce4285244a5f299bc0b337af579c6b6f3833984ba04d: Status 404 returned error can't find the container with id e882e1575c6535cef951ce4285244a5f299bc0b337af579c6b6f3833984ba04d
Sep 10 13:21:47.763220 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:47.763188    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d54b58_eea2_4440_b09f_6445e29204e4.slice/crio-974dececa893c07da84ebed70f0902c6e92d8542b5912b9b28d722b36f58bc88 WatchSource:0}: Error finding container 974dececa893c07da84ebed70f0902c6e92d8542b5912b9b28d722b36f58bc88: Status 404 returned error can't find the container with id 974dececa893c07da84ebed70f0902c6e92d8542b5912b9b28d722b36f58bc88
Sep 10 13:21:47.811651 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.811595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vs6\" (UniqueName: \"kubernetes.io/projected/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-api-access-r7vs6\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:47.813838 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.813816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vs6\" (UniqueName: \"kubernetes.io/projected/d2dd98c8-d5d5-422a-a507-1480a9609412-kube-api-access-r7vs6\") pod \"insights-runtime-extractor-rnmmb\" (UID: \"d2dd98c8-d5d5-422a-a507-1480a9609412\") " pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:47.922969 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.922895    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-576868dd66-clmnw"]
Sep 10 13:21:47.926085 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:47.926064    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f9a8c0_7e20_40de_b964_0ba36aab5ea0.slice/crio-6763903e3cfd0e8677faafdf1820408bc5fc8403137d740fd9409d6696ff4f1e WatchSource:0}: Error finding container 6763903e3cfd0e8677faafdf1820408bc5fc8403137d740fd9409d6696ff4f1e: Status 404 returned error can't find the container with id 6763903e3cfd0e8677faafdf1820408bc5fc8403137d740fd9409d6696ff4f1e
Sep 10 13:21:47.926828 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.926812    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rnmmb"
Sep 10 13:21:47.987063 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.986686    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-62tzd" event={"ID":"c9e7452e-8ac7-40b2-a170-fb36f5f4cff9","Type":"ContainerStarted","Data":"3904219a39805c0308f073b607a56ae1af9e0eba2711402d6d490bee84c76bee"}
Sep 10 13:21:47.988097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.988073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ch54g" event={"ID":"63ac7b44-01ac-4815-8cb8-fa1af93a57f7","Type":"ContainerStarted","Data":"e882e1575c6535cef951ce4285244a5f299bc0b337af579c6b6f3833984ba04d"}
Sep 10 13:21:47.989510 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.989490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-llj8p" event={"ID":"f64e39f7-de8e-43ea-987a-02c4811d7050","Type":"ContainerStarted","Data":"a84920b7a0bdbc4c92a95e0f86b30986ae74b5ecb89567d2b881d5237353dcb7"}
Sep 10 13:21:47.989602 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.989543    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:21:47.990628 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.990607    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6976s" event={"ID":"14d54b58-eea2-4440-b09f-6445e29204e4","Type":"ContainerStarted","Data":"974dececa893c07da84ebed70f0902c6e92d8542b5912b9b28d722b36f58bc88"}
Sep 10 13:21:47.991722 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.991701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm" event={"ID":"f87cd7ce-82ce-4272-b867-47107fbaa79d","Type":"ContainerStarted","Data":"0f30ccf639ebaba40111040ad96b39e1a5809283d1cb0a890d01e6e435d7f658"}
Sep 10 13:21:47.993005 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.992984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-576868dd66-clmnw" event={"ID":"36f9a8c0-7e20-40de-b964-0ba36aab5ea0","Type":"ContainerStarted","Data":"3ca6039a6495a8d9ab894cb226e5cd441632f92fd42d8435ca9079f588832c6c"}
Sep 10 13:21:47.993085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.993020    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-576868dd66-clmnw" event={"ID":"36f9a8c0-7e20-40de-b964-0ba36aab5ea0","Type":"ContainerStarted","Data":"6763903e3cfd0e8677faafdf1820408bc5fc8403137d740fd9409d6696ff4f1e"}
Sep 10 13:21:47.993121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:47.993113    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:21:48.041252 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:48.041185    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-62tzd" podStartSLOduration=75.389214387 podStartE2EDuration="1m20.041167071s" totalImagesPullingTime="4.651952684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:48.040725421 +0000 UTC m=+213.057557392" watchObservedRunningTime="2026-09-10 13:21:48.041167071 +0000 UTC m=+213.057999042"
Sep 10 13:21:48.130325 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:48.130158    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-llj8p" podStartSLOduration=75.513850422 podStartE2EDuration="1m20.130137417s" totalImagesPullingTime="4.616286995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:48.129427143 +0000 UTC m=+213.146259126" watchObservedRunningTime="2026-09-10 13:21:48.130137417 +0000 UTC m=+213.146969389"
Sep 10 13:21:48.130834 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:48.130802    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-576868dd66-clmnw" podStartSLOduration=3.130790988 podStartE2EDuration="3.130790988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:48.09353492 +0000 UTC m=+213.110366890" watchObservedRunningTime="2026-09-10 13:21:48.130790988 +0000 UTC m=+213.147622959"
Sep 10 13:21:48.287563 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:48.287516    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rnmmb"]
Sep 10 13:21:48.291277 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:48.291240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2dd98c8_d5d5_422a_a507_1480a9609412.slice/crio-9c6c46efa47ce48d5975184bd17b0c0f1e16a3774fc8d6da2f4f950267991fa1 WatchSource:0}: Error finding container 9c6c46efa47ce48d5975184bd17b0c0f1e16a3774fc8d6da2f4f950267991fa1: Status 404 returned error can't find the container with id 9c6c46efa47ce48d5975184bd17b0c0f1e16a3774fc8d6da2f4f950267991fa1
Sep 10 13:21:49.002345 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:49.002304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rnmmb" event={"ID":"d2dd98c8-d5d5-422a-a507-1480a9609412","Type":"ContainerStarted","Data":"a5fd189c882c36919e36e424a1c9dc47d0bb26afd6518f8e1a2f918b7e789539"}
Sep 10 13:21:49.002456 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:49.002404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rnmmb" event={"ID":"d2dd98c8-d5d5-422a-a507-1480a9609412","Type":"ContainerStarted","Data":"9c6c46efa47ce48d5975184bd17b0c0f1e16a3774fc8d6da2f4f950267991fa1"}
Sep 10 13:21:50.007653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.007618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm" event={"ID":"f87cd7ce-82ce-4272-b867-47107fbaa79d","Type":"ContainerStarted","Data":"9bc0a4ce84af1324d66c98c96eccbe6e039d3aed1b8cb4c63a5ebdb5ae7057b2"}
Sep 10 13:21:50.008088 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.007885    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"
Sep 10 13:21:50.013322 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.013297    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm"
Sep 10 13:21:50.034524 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.034474    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-85hmm" podStartSLOduration=3.745978606 podStartE2EDuration="5.034457095s" totalImagesPullingTime="1.288478489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:50.034283242 +0000 UTC m=+215.051115214" watchObservedRunningTime="2026-09-10 13:21:50.034457095 +0000 UTC m=+215.051289067"
Sep 10 13:21:50.382867 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.382834    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"]
Sep 10 13:21:50.385034 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.385018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.394471 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.394448    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 13:21:50.403827 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.403253    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 13:21:50.403827 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.403550    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 13:21:50.404532 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.404102    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 13:21:50.404532 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.404117    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 13:21:50.404532 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.404359    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-z9cn5\""
Sep 10 13:21:50.430490 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.430464    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"]
Sep 10 13:21:50.434147 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.434118    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-782v4\" (UniqueName: \"kubernetes.io/projected/05797d67-a8e9-418a-807a-8f6b521fa0f1-kube-api-access-782v4\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.434390 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.434157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05797d67-a8e9-418a-807a-8f6b521fa0f1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.434390 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.434177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05797d67-a8e9-418a-807a-8f6b521fa0f1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.434390 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.434237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05797d67-a8e9-418a-807a-8f6b521fa0f1-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.535167 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.535134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-782v4\" (UniqueName: \"kubernetes.io/projected/05797d67-a8e9-418a-807a-8f6b521fa0f1-kube-api-access-782v4\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.535308 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.535190    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05797d67-a8e9-418a-807a-8f6b521fa0f1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.535308 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.535237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05797d67-a8e9-418a-807a-8f6b521fa0f1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.535308 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.535281    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05797d67-a8e9-418a-807a-8f6b521fa0f1-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.535533 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:50.535517    2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 10 13:21:50.535594 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:50.535580    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05797d67-a8e9-418a-807a-8f6b521fa0f1-prometheus-operator-tls podName:05797d67-a8e9-418a-807a-8f6b521fa0f1 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:51.035560293 +0000 UTC m=+216.052392243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/05797d67-a8e9-418a-807a-8f6b521fa0f1-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-2fdjx" (UID: "05797d67-a8e9-418a-807a-8f6b521fa0f1") : secret "prometheus-operator-tls" not found
Sep 10 13:21:50.536071 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.536048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05797d67-a8e9-418a-807a-8f6b521fa0f1-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.537953 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.537931    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05797d67-a8e9-418a-807a-8f6b521fa0f1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:50.551360 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:50.551314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-782v4\" (UniqueName: \"kubernetes.io/projected/05797d67-a8e9-418a-807a-8f6b521fa0f1-kube-api-access-782v4\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:51.013829 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.013791    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rnmmb" event={"ID":"d2dd98c8-d5d5-422a-a507-1480a9609412","Type":"ContainerStarted","Data":"ca6ec492062c3b740d598381f3e49712f464115b497e3d5a7e8ae7822b47a163"}
Sep 10 13:21:51.015585 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.015553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6976s" event={"ID":"14d54b58-eea2-4440-b09f-6445e29204e4","Type":"ContainerStarted","Data":"933a08f7e74e0ecdb568f47b9c9edda6f1476889ea2ecc5272d29f6d8344a7b2"}
Sep 10 13:21:51.017732 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.017709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ch54g" event={"ID":"63ac7b44-01ac-4815-8cb8-fa1af93a57f7","Type":"ContainerStarted","Data":"e7263609e11a4138212b7c8a02a5aedabef27ab26660d66b5c6433f45d3a150e"}
Sep 10 13:21:51.017842 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.017763    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ch54g" event={"ID":"63ac7b44-01ac-4815-8cb8-fa1af93a57f7","Type":"ContainerStarted","Data":"c0ed69979c7d657763a7096c4be560016fb650f174f68c7e57e8116b24052053"}
Sep 10 13:21:51.039427 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.039391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05797d67-a8e9-418a-807a-8f6b521fa0f1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:51.042743 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.042715    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05797d67-a8e9-418a-807a-8f6b521fa0f1-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2fdjx\" (UID: \"05797d67-a8e9-418a-807a-8f6b521fa0f1\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:51.078151 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.078087    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ch54g" podStartSLOduration=3.467730916 podStartE2EDuration="6.078064103s" totalImagesPullingTime="2.610333187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:51.076857273 +0000 UTC m=+216.093689245" watchObservedRunningTime="2026-09-10 13:21:51.078064103 +0000 UTC m=+216.094896077"
Sep 10 13:21:51.078923 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.078866    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6976s" podStartSLOduration=3.450641676 podStartE2EDuration="6.078854209s" totalImagesPullingTime="2.628212533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:51.049717561 +0000 UTC m=+216.066549533" watchObservedRunningTime="2026-09-10 13:21:51.078854209 +0000 UTC m=+216.095686183"
Sep 10 13:21:51.328840 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.328759    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"
Sep 10 13:21:51.630223 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:51.630131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2fdjx"]
Sep 10 13:21:51.780512 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:51.780478    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05797d67_a8e9_418a_807a_8f6b521fa0f1.slice/crio-71daca7142c5dc18ca0894ce413fbfdd7b53a283089c316cf6130f299017a4fa WatchSource:0}: Error finding container 71daca7142c5dc18ca0894ce413fbfdd7b53a283089c316cf6130f299017a4fa: Status 404 returned error can't find the container with id 71daca7142c5dc18ca0894ce413fbfdd7b53a283089c316cf6130f299017a4fa
Sep 10 13:21:52.022517 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:52.022484    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rnmmb" event={"ID":"d2dd98c8-d5d5-422a-a507-1480a9609412","Type":"ContainerStarted","Data":"c0e2cd6e4c424d4ead579389226b55ccd8e3cd1e9fb32d80180df063ddc10c0c"}
Sep 10 13:21:52.023689 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:52.023663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx" event={"ID":"05797d67-a8e9-418a-807a-8f6b521fa0f1","Type":"ContainerStarted","Data":"71daca7142c5dc18ca0894ce413fbfdd7b53a283089c316cf6130f299017a4fa"}
Sep 10 13:21:52.023862 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:52.023846    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ch54g"
Sep 10 13:21:52.057274 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:52.057224    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rnmmb" podStartSLOduration=3.706964499 podStartE2EDuration="7.057190536s" totalImagesPullingTime="3.350226037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:21:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:52.055536098 +0000 UTC m=+217.072368069" watchObservedRunningTime="2026-09-10 13:21:52.057190536 +0000 UTC m=+217.074022506"
Sep 10 13:21:54.033039 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:54.033006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx" event={"ID":"05797d67-a8e9-418a-807a-8f6b521fa0f1","Type":"ContainerStarted","Data":"7adc4ac7068ec8ed7f1f855267991f314ded3aeff8dc2a4b3b210a3dee6a6b9b"}
Sep 10 13:21:54.033428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:54.033054    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx" event={"ID":"05797d67-a8e9-418a-807a-8f6b521fa0f1","Type":"ContainerStarted","Data":"9fd5e3c0d2187892b00a6983f7830541d44a2d521f5c57e480556cffc0466fa5"}
Sep 10 13:21:54.068959 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:54.068906    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-2fdjx" podStartSLOduration=2.7415843730000002 podStartE2EDuration="4.068890252s" totalImagesPullingTime="1.327305879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:54.068384852 +0000 UTC m=+219.085216821" watchObservedRunningTime="2026-09-10 13:21:54.068890252 +0000 UTC m=+219.085722249"
Sep 10 13:21:56.089579 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.089543    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"]
Sep 10 13:21:56.092026 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.092006    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"
Sep 10 13:21:56.104814 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.104790    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cw8zw\""
Sep 10 13:21:56.104941 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.104835    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 13:21:56.106523 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.106507    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 13:21:56.106976 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.106962    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 13:21:56.111425 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.111408    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 13:21:56.125900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.125878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"]
Sep 10 13:21:56.164912 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.164872    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"]
Sep 10 13:21:56.166766 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.166751    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.171975 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.171913    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 13:21:56.175999 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.175978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlbp\" (UniqueName: \"kubernetes.io/projected/9adbcc02-3381-45e3-9134-920a08f4ecc6-kube-api-access-shlbp\") pod \"managed-serviceaccount-addon-agent-57fbb5d868-hh92s\" (UID: \"9adbcc02-3381-45e3-9134-920a08f4ecc6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"
Sep 10 13:21:56.176074 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.176028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9adbcc02-3381-45e3-9134-920a08f4ecc6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57fbb5d868-hh92s\" (UID: \"9adbcc02-3381-45e3-9134-920a08f4ecc6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"
Sep 10 13:21:56.185150 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.185125    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"]
Sep 10 13:21:56.187144 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.187130    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.190529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.190510    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 13:21:56.191770 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.191750    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8xknc\""
Sep 10 13:21:56.192738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.192719    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"]
Sep 10 13:21:56.196847 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.196830    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 13:21:56.233964 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.233936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"]
Sep 10 13:21:56.240487 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.240462    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-srmm7"]
Sep 10 13:21:56.242548 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.242532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.249240 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.249220    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 13:21:56.250026 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.250009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 13:21:56.250117 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.250106    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vt7d7\""
Sep 10 13:21:56.250188 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.250175    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 13:21:56.271281 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.271248    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-h827l"]
Sep 10 13:21:56.273643 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.273627    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"]
Sep 10 13:21:56.273779 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.273762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.275555 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.275539    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.276325 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hpd2\" (UniqueName: \"kubernetes.io/projected/2929e2a9-8338-4c94-aaec-df9349da998b-kube-api-access-4hpd2\") pod \"klusterlet-addon-workmgr-5687f77958-px9hm\" (UID: \"2929e2a9-8338-4c94-aaec-df9349da998b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.276419 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.276419 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-accelerators-collector-config\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.276419 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9adbcc02-3381-45e3-9134-920a08f4ecc6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57fbb5d868-hh92s\" (UID: \"9adbcc02-3381-45e3-9134-920a08f4ecc6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"
Sep 10 13:21:56.276577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.276577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.276577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2929e2a9-8338-4c94-aaec-df9349da998b-tmp\") pod \"klusterlet-addon-workmgr-5687f77958-px9hm\" (UID: \"2929e2a9-8338-4c94-aaec-df9349da998b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.276577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2n2\" (UniqueName: \"kubernetes.io/projected/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-kube-api-access-fp2n2\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.276728 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276628    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1130036-b991-491a-b56c-4717d21e4023-sys\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.276728 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-tls\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.276799 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2929e2a9-8338-4c94-aaec-df9349da998b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5687f77958-px9hm\" (UID: \"2929e2a9-8338-4c94-aaec-df9349da998b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.276799 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlbp\" (UniqueName: \"kubernetes.io/projected/9adbcc02-3381-45e3-9134-920a08f4ecc6-kube-api-access-shlbp\") pod \"managed-serviceaccount-addon-agent-57fbb5d868-hh92s\" (UID: \"9adbcc02-3381-45e3-9134-920a08f4ecc6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"
Sep 10 13:21:56.276896 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.276896 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-textfile\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.276896 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276870    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75w5d\" (UniqueName: \"kubernetes.io/projected/c1130036-b991-491a-b56c-4717d21e4023-kube-api-access-75w5d\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.277016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276911    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c1130036-b991-491a-b56c-4717d21e4023-root\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.277016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-wtmp\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.277016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.276972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1130036-b991-491a-b56c-4717d21e4023-metrics-client-ca\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.277503 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:56.277479    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-tls\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret"
Sep 10 13:21:56.278841 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.278825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9adbcc02-3381-45e3-9134-920a08f4ecc6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-57fbb5d868-hh92s\" (UID: \"9adbcc02-3381-45e3-9134-920a08f4ecc6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"
Sep 10 13:21:56.280022 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:56.279980    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-dockercfg-nrkh7\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nrkh7\"" type="*v1.Secret"
Sep 10 13:21:56.280022 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:56.280005    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-state-metrics-custom-resource-state-configmap\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap"
Sep 10 13:21:56.280170 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:56.280149    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-state-metrics-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret"
Sep 10 13:21:56.280278 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:56.280257    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-service-proxy-server-certificates\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret"
Sep 10 13:21:56.280401 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:56.280384    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret"
Sep 10 13:21:56.280401 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:56.280379    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret"
Sep 10 13:21:56.280401 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:56.280384    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-ca\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret"
Sep 10 13:21:56.301526 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.301491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlbp\" (UniqueName: \"kubernetes.io/projected/9adbcc02-3381-45e3-9134-920a08f4ecc6-kube-api-access-shlbp\") pod \"managed-serviceaccount-addon-agent-57fbb5d868-hh92s\" (UID: \"9adbcc02-3381-45e3-9134-920a08f4ecc6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"
Sep 10 13:21:56.304396 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.304372    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-h827l"]
Sep 10 13:21:56.310665 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.310643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"]
Sep 10 13:21:56.377710 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377629    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2n2\" (UniqueName: \"kubernetes.io/projected/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-kube-api-access-fp2n2\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.377710 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwpc2\" (UniqueName: \"kubernetes.io/projected/0a666639-dcc0-468a-91d6-1034fd5ebdad-kube-api-access-jwpc2\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.377710 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1130036-b991-491a-b56c-4717d21e4023-sys\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.377710 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-tls\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1130036-b991-491a-b56c-4717d21e4023-sys\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2929e2a9-8338-4c94-aaec-df9349da998b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5687f77958-px9hm\" (UID: \"2929e2a9-8338-4c94-aaec-df9349da998b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8842d11-4ed0-401f-9cd4-9479765035a3-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-ca\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377876    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377933    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.377978 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377972    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a666639-dcc0-468a-91d6-1034fd5ebdad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.377999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-textfile\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378015    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75w5d\" (UniqueName: \"kubernetes.io/projected/c1130036-b991-491a-b56c-4717d21e4023-kube-api-access-75w5d\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c1130036-b991-491a-b56c-4717d21e4023-root\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-wtmp\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1130036-b991-491a-b56c-4717d21e4023-metrics-client-ca\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wc7\" (UniqueName: \"kubernetes.io/projected/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-api-access-89wc7\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c1130036-b991-491a-b56c-4717d21e4023-root\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hpd2\" (UniqueName: \"kubernetes.io/projected/2929e2a9-8338-4c94-aaec-df9349da998b-kube-api-access-4hpd2\") pod \"klusterlet-addon-workmgr-5687f77958-px9hm\" (UID: \"2929e2a9-8338-4c94-aaec-df9349da998b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-accelerators-collector-config\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-wtmp\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378284    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-hub\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.378447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-textfile\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.379190 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2929e2a9-8338-4c94-aaec-df9349da998b-tmp\") pod \"klusterlet-addon-workmgr-5687f77958-px9hm\" (UID: \"2929e2a9-8338-4c94-aaec-df9349da998b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.379190 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e8842d11-4ed0-401f-9cd4-9479765035a3-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.379190 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.378700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1130036-b991-491a-b56c-4717d21e4023-metrics-client-ca\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.379190 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.379087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2929e2a9-8338-4c94-aaec-df9349da998b-tmp\") pod \"klusterlet-addon-workmgr-5687f77958-px9hm\" (UID: \"2929e2a9-8338-4c94-aaec-df9349da998b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.379190 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.379158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.379456 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.379218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-accelerators-collector-config\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.380414 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.380390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.380660 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.380642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c1130036-b991-491a-b56c-4717d21e4023-node-exporter-tls\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.380874 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.380852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.380973 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.380895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2929e2a9-8338-4c94-aaec-df9349da998b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5687f77958-px9hm\" (UID: \"2929e2a9-8338-4c94-aaec-df9349da998b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.381051 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.381033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.406577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.406549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75w5d\" (UniqueName: \"kubernetes.io/projected/c1130036-b991-491a-b56c-4717d21e4023-kube-api-access-75w5d\") pod \"node-exporter-srmm7\" (UID: \"c1130036-b991-491a-b56c-4717d21e4023\") " pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.408447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.408431    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"
Sep 10 13:21:56.411536 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.411512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2n2\" (UniqueName: \"kubernetes.io/projected/89a8fe67-1f1e-423b-ab4b-0465f9bbcb64-kube-api-access-fp2n2\") pod \"openshift-state-metrics-657fd6c54-zm9th\" (UID: \"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.415932 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.415911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hpd2\" (UniqueName: \"kubernetes.io/projected/2929e2a9-8338-4c94-aaec-df9349da998b-kube-api-access-4hpd2\") pod \"klusterlet-addon-workmgr-5687f77958-px9hm\" (UID: \"2929e2a9-8338-4c94-aaec-df9349da998b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.446445 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:56.446400    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=setup pod=kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator(24e19692c959ce3358ef1453cec7edef)\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podUID="24e19692c959ce3358ef1453cec7edef"
Sep 10 13:21:56.475990 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.475651    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:21:56.479598 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.479567    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.479678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.479616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a666639-dcc0-468a-91d6-1034fd5ebdad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.479678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.479658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wc7\" (UniqueName: \"kubernetes.io/projected/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-api-access-89wc7\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.479807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.479857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-hub\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.479898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e8842d11-4ed0-401f-9cd4-9479765035a3-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.479925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwpc2\" (UniqueName: \"kubernetes.io/projected/0a666639-dcc0-468a-91d6-1034fd5ebdad-kube-api-access-jwpc2\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.479964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.479994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8842d11-4ed0-401f-9cd4-9479765035a3-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.480020    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.480047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-ca\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.480073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.480593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.480532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/e8842d11-4ed0-401f-9cd4-9479765035a3-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.481156 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.480701    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a666639-dcc0-468a-91d6-1034fd5ebdad-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.481156 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.480974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8842d11-4ed0-401f-9cd4-9479765035a3-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.496601 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.496572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"
Sep 10 13:21:56.502374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.502344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwpc2\" (UniqueName: \"kubernetes.io/projected/0a666639-dcc0-468a-91d6-1034fd5ebdad-kube-api-access-jwpc2\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:56.502517 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.502347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wc7\" (UniqueName: \"kubernetes.io/projected/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-api-access-89wc7\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:56.551523 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.551473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-srmm7"
Sep 10 13:21:56.566224 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:56.565947    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1130036_b991_491a_b56c_4717d21e4023.slice/crio-9cb4610d40a819d519d96172cc463c188deed2c79f9acfc2d6675ef2fb0f845a WatchSource:0}: Error finding container 9cb4610d40a819d519d96172cc463c188deed2c79f9acfc2d6675ef2fb0f845a: Status 404 returned error can't find the container with id 9cb4610d40a819d519d96172cc463c188deed2c79f9acfc2d6675ef2fb0f845a
Sep 10 13:21:56.688935 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.688913    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s"]
Sep 10 13:21:56.690913 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:56.690888    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9adbcc02_3381_45e3_9134_920a08f4ecc6.slice/crio-52026318507f30554e065c380630471db9ce2bd71a0710783bd2c79c989a2b2b WatchSource:0}: Error finding container 52026318507f30554e065c380630471db9ce2bd71a0710783bd2c79c989a2b2b: Status 404 returned error can't find the container with id 52026318507f30554e065c380630471db9ce2bd71a0710783bd2c79c989a2b2b
Sep 10 13:21:56.778221 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.778182    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"]
Sep 10 13:21:56.780358 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:56.780334    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2929e2a9_8338_4c94_aaec_df9349da998b.slice/crio-8a1dff711d7e2ba417e3bd34beb9f14e8cd71f60cea4c4da3ef46a2f5cb9e1be WatchSource:0}: Error finding container 8a1dff711d7e2ba417e3bd34beb9f14e8cd71f60cea4c4da3ef46a2f5cb9e1be: Status 404 returned error can't find the container with id 8a1dff711d7e2ba417e3bd34beb9f14e8cd71f60cea4c4da3ef46a2f5cb9e1be
Sep 10 13:21:56.803247 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:56.803175    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th"]
Sep 10 13:21:56.805476 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:56.805445    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a8fe67_1f1e_423b_ab4b_0465f9bbcb64.slice/crio-6c01d38f3638f6f901c3c36d319ae9e48977e4e27aa9ab0cfa87466213e4aa79 WatchSource:0}: Error finding container 6c01d38f3638f6f901c3c36d319ae9e48977e4e27aa9ab0cfa87466213e4aa79: Status 404 returned error can't find the container with id 6c01d38f3638f6f901c3c36d319ae9e48977e4e27aa9ab0cfa87466213e4aa79
Sep 10 13:21:57.044853 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.044822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th" event={"ID":"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64","Type":"ContainerStarted","Data":"d04f1001c94f459576c0a4b56f126d685100e8a70405262cf9ae173f2e462280"}
Sep 10 13:21:57.045013 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.044885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th" event={"ID":"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64","Type":"ContainerStarted","Data":"d3d63d8a3cf9a91d7283fe327e61bfcf56048fa14b3906d860728b12a4f7da04"}
Sep 10 13:21:57.045013 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.044907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th" event={"ID":"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64","Type":"ContainerStarted","Data":"6c01d38f3638f6f901c3c36d319ae9e48977e4e27aa9ab0cfa87466213e4aa79"}
Sep 10 13:21:57.045863 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.045842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm" event={"ID":"2929e2a9-8338-4c94-aaec-df9349da998b","Type":"ContainerStarted","Data":"8a1dff711d7e2ba417e3bd34beb9f14e8cd71f60cea4c4da3ef46a2f5cb9e1be"}
Sep 10 13:21:57.046948 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.046926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s" event={"ID":"9adbcc02-3381-45e3-9134-920a08f4ecc6","Type":"ContainerStarted","Data":"52026318507f30554e065c380630471db9ce2bd71a0710783bd2c79c989a2b2b"}
Sep 10 13:21:57.048012 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.047993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-srmm7" event={"ID":"c1130036-b991-491a-b56c-4717d21e4023","Type":"ContainerStarted","Data":"9cb4610d40a819d519d96172cc463c188deed2c79f9acfc2d6675ef2fb0f845a"}
Sep 10 13:21:57.115302 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.115268    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 10 13:21:57.120799 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.120777    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:57.180441 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.180404    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 13:21:57.188001 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.187974    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 13:21:57.192870 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.192846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:57.193100 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.193076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:57.392448 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.392363    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 13:21:57.403483 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.403441    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-hub\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:57.480427 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:57.480397    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:57.480427 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:57.480415    2578 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-ca: failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:57.480653 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:57.480417    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:57.480653 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:57.480480    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-kube-rbac-proxy-config podName:e8842d11-4ed0-401f-9cd4-9479765035a3 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:57.980460043 +0000 UTC m=+222.997292106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-kube-rbac-proxy-config") pod "kube-state-metrics-74cb57df7-h827l" (UID: "e8842d11-4ed0-401f-9cd4-9479765035a3") : failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:57.480653 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:57.480496    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-ca podName:0a666639-dcc0-468a-91d6-1034fd5ebdad nodeName:}" failed. No retries permitted until 2026-09-10 13:21:57.980489169 +0000 UTC m=+222.997321118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ca" (UniqueName: "kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-ca") pod "cluster-proxy-proxy-agent-6b9b6b6757-zsdtw" (UID: "0a666639-dcc0-468a-91d6-1034fd5ebdad") : failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:57.480653 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:21:57.480511    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-tls podName:e8842d11-4ed0-401f-9cd4-9479765035a3 nodeName:}" failed. No retries permitted until 2026-09-10 13:21:57.980503009 +0000 UTC m=+222.997334960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-h827l" (UID: "e8842d11-4ed0-401f-9cd4-9479765035a3") : failed to sync secret cache: timed out waiting for the condition
Sep 10 13:21:57.514884 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.514847    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nrkh7\""
Sep 10 13:21:57.708087 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.708062    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 13:21:57.756414 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.756393    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 10 13:21:57.865503 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.865469    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 10 13:21:57.990867 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.990821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:57.991135 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.990875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-ca\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:57.991135 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.990905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:57.993859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.993816    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:57.994303 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.994258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8842d11-4ed0-401f-9cd4-9479765035a3-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-h827l\" (UID: \"e8842d11-4ed0-401f-9cd4-9479765035a3\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:57.994987 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:57.994922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0a666639-dcc0-468a-91d6-1034fd5ebdad-ca\") pod \"cluster-proxy-proxy-agent-6b9b6b6757-zsdtw\" (UID: \"0a666639-dcc0-468a-91d6-1034fd5ebdad\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:58.054890 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:58.054786    2578 generic.go:356] "Generic (PLEG): container finished" podID="c1130036-b991-491a-b56c-4717d21e4023" containerID="a351c8139501c7b33d6e7646dc5f58e59f4ed47b8025d0c7cae417adbdbb1e8b" exitCode=0
Sep 10 13:21:58.055072 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:58.054895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-srmm7" event={"ID":"c1130036-b991-491a-b56c-4717d21e4023","Type":"ContainerDied","Data":"a351c8139501c7b33d6e7646dc5f58e59f4ed47b8025d0c7cae417adbdbb1e8b"}
Sep 10 13:21:58.087610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:58.087572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l"
Sep 10 13:21:58.092156 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:58.092089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"
Sep 10 13:21:58.636355 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:58.636275    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-h827l"]
Sep 10 13:21:58.643257 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:58.643222    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw"]
Sep 10 13:21:58.647626 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:58.647597    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8842d11_4ed0_401f_9cd4_9479765035a3.slice/crio-24192291858dba78c4f20763c7e702e38029b38045342880a138b97c20c2ebae WatchSource:0}: Error finding container 24192291858dba78c4f20763c7e702e38029b38045342880a138b97c20c2ebae: Status 404 returned error can't find the container with id 24192291858dba78c4f20763c7e702e38029b38045342880a138b97c20c2ebae
Sep 10 13:21:58.648403 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:21:58.648375    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a666639_dcc0_468a_91d6_1034fd5ebdad.slice/crio-721e214495fd5970ed026e03fd7fdd7b4e8d269d9079045ccf14ba6afc1a2490 WatchSource:0}: Error finding container 721e214495fd5970ed026e03fd7fdd7b4e8d269d9079045ccf14ba6afc1a2490: Status 404 returned error can't find the container with id 721e214495fd5970ed026e03fd7fdd7b4e8d269d9079045ccf14ba6afc1a2490
Sep 10 13:21:59.059546 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:59.059509    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw" event={"ID":"0a666639-dcc0-468a-91d6-1034fd5ebdad","Type":"ContainerStarted","Data":"721e214495fd5970ed026e03fd7fdd7b4e8d269d9079045ccf14ba6afc1a2490"}
Sep 10 13:21:59.061895 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:59.061862    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th" event={"ID":"89a8fe67-1f1e-423b-ab4b-0465f9bbcb64","Type":"ContainerStarted","Data":"25e5a7b58a5bf59dae8d3c9c00a8fb1e6587656b0a68bd9d7465440b659f8e8d"}
Sep 10 13:21:59.063323 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:59.063299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l" event={"ID":"e8842d11-4ed0-401f-9cd4-9479765035a3","Type":"ContainerStarted","Data":"24192291858dba78c4f20763c7e702e38029b38045342880a138b97c20c2ebae"}
Sep 10 13:21:59.065536 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:59.065503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-srmm7" event={"ID":"c1130036-b991-491a-b56c-4717d21e4023","Type":"ContainerStarted","Data":"a398c17f75204387f2011e7d4f75483c97fd3b6ee07e3e1e22660ad65cc7b12c"}
Sep 10 13:21:59.065636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:59.065548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-srmm7" event={"ID":"c1130036-b991-491a-b56c-4717d21e4023","Type":"ContainerStarted","Data":"4c196c2f449cb51e8c5f90906f6b97808ef17b7406f723da454d3c43ce48c6ff"}
Sep 10 13:21:59.145878 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:59.145817    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-zm9th" podStartSLOduration=1.706285803 podStartE2EDuration="3.145796042s" totalImagesPullingTime="1.439510239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:59.107448672 +0000 UTC m=+224.124280642" watchObservedRunningTime="2026-09-10 13:21:59.145796042 +0000 UTC m=+224.162628015"
Sep 10 13:21:59.146183 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:21:59.146156    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-srmm7" podStartSLOduration=2.025124742 podStartE2EDuration="3.146148024s" totalImagesPullingTime="1.060603892s" totalInitContainerRuntime="60.41939ms" isStatefulPod=true podCreationTimestamp="2026-09-10 13:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:21:59.145220172 +0000 UTC m=+224.162052144" watchObservedRunningTime="2026-09-10 13:21:59.146148024 +0000 UTC m=+224.162979993"
Sep 10 13:22:00.073703 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:00.073534    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s" event={"ID":"9adbcc02-3381-45e3-9134-920a08f4ecc6","Type":"ContainerStarted","Data":"d80803c80d66e18e1159c46c04ba7b49f9fe9aa4814d9126985498fdb2132250"}
Sep 10 13:22:00.104897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:00.104845    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-57fbb5d868-hh92s" podStartSLOduration=1.031052971 podStartE2EDuration="4.104830426s" totalImagesPullingTime="3.073777455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:00.103548261 +0000 UTC m=+225.120380246" watchObservedRunningTime="2026-09-10 13:22:00.104830426 +0000 UTC m=+225.121662396"
Sep 10 13:22:01.081082 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:01.081048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l" event={"ID":"e8842d11-4ed0-401f-9cd4-9479765035a3","Type":"ContainerStarted","Data":"89773988cf128ad8f1e09af6a8e00e9ce833736aecebc52eaa445214c3a35973"}
Sep 10 13:22:01.081553 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:01.081102    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l" event={"ID":"e8842d11-4ed0-401f-9cd4-9479765035a3","Type":"ContainerStarted","Data":"2b1dca306ca675c713e5ca7892c4d90837360052e9626b18753962aa11b69cea"}
Sep 10 13:22:01.081553 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:01.081122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l" event={"ID":"e8842d11-4ed0-401f-9cd4-9479765035a3","Type":"ContainerStarted","Data":"9cd65bad2400667f2623050f478e251f1b247dea32a686457dfe0f6b8ab7b3ac"}
Sep 10 13:22:01.126496 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:01.126437    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-h827l" podStartSLOduration=3.473892295 podStartE2EDuration="5.126416738s" totalImagesPullingTime="1.652524443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:01.123028582 +0000 UTC m=+226.139860553" watchObservedRunningTime="2026-09-10 13:22:01.126416738 +0000 UTC m=+226.143248709"
Sep 10 13:22:02.030171 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:02.030136    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ch54g"
Sep 10 13:22:03.092241 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:03.092188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw" event={"ID":"0a666639-dcc0-468a-91d6-1034fd5ebdad","Type":"ContainerStarted","Data":"3d7e068e3d33e0de0200a8e6abd1ca1600b6d84d28ec8a6bdeb88151558bb727"}
Sep 10 13:22:03.092241 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:03.092247    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw" event={"ID":"0a666639-dcc0-468a-91d6-1034fd5ebdad","Type":"ContainerStarted","Data":"0aebd624d00213cb019cfeb324d2dab21f0cc35610c395cef2159d3fa783b1f8"}
Sep 10 13:22:03.092773 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:03.092260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw" event={"ID":"0a666639-dcc0-468a-91d6-1034fd5ebdad","Type":"ContainerStarted","Data":"d6d244fe41eedf3ea96c6347c7185ec6b7a93543ee419e0682a4d80505ade920"}
Sep 10 13:22:03.140677 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:03.140628    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b9b6b6757-zsdtw" podStartSLOduration=3.220865919 podStartE2EDuration="7.140608233s" totalImagesPullingTime="3.919742314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:03.140023172 +0000 UTC m=+228.156855144" watchObservedRunningTime="2026-09-10 13:22:03.140608233 +0000 UTC m=+228.157440204"
Sep 10 13:22:08.116055 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:08.115975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm" event={"ID":"2929e2a9-8338-4c94-aaec-df9349da998b","Type":"ContainerStarted","Data":"d7fc2ef3225cb2f761624f97349837b5a7591983de2ca73719a46fe4b3f0ad78"}
Sep 10 13:22:08.116453 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:08.116176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:22:08.117731 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:08.117703    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm"
Sep 10 13:22:08.151106 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:08.151053    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5687f77958-px9hm" podStartSLOduration=1.161088347 podStartE2EDuration="12.151040629s" totalImagesPullingTime="10.989952282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:08.150132933 +0000 UTC m=+233.166964916" watchObservedRunningTime="2026-09-10 13:22:08.151040629 +0000 UTC m=+233.167872600"
Sep 10 13:22:09.006777 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:09.006751    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-576868dd66-clmnw"
Sep 10 13:22:10.126284 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:10.126257    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_24e19692c959ce3358ef1453cec7edef/setup/4.log"
Sep 10 13:22:10.126681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:10.126292    2578 generic.go:356] "Generic (PLEG): container finished" podID="24e19692c959ce3358ef1453cec7edef" containerID="8d0c43445c52d486e915a10435bf1c4ac9fc03c94ce42d7b9c87922906cbfe8e" exitCode=0
Sep 10 13:22:10.126681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:10.126374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerDied","Data":"8d0c43445c52d486e915a10435bf1c4ac9fc03c94ce42d7b9c87922906cbfe8e"}
Sep 10 13:22:10.126681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:10.126432    2578 scope.go:117] "RemoveContainer" containerID="f227eae44fb7abb0d51682b3ca9423f8b89afb8096958572a6b3adbfcb20cf46"
Sep 10 13:22:10.126835 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:10.126738    2578 scope.go:117] "RemoveContainer" containerID="f227eae44fb7abb0d51682b3ca9423f8b89afb8096958572a6b3adbfcb20cf46"
Sep 10 13:22:10.136783 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:22:10.136678    2578 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_4 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: 'f227eae44fb7abb0d51682b3ca9423f8b89afb8096958572a6b3adbfcb20cf46'" containerID="f227eae44fb7abb0d51682b3ca9423f8b89afb8096958572a6b3adbfcb20cf46"
Sep 10 13:22:10.136783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:10.136717    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f227eae44fb7abb0d51682b3ca9423f8b89afb8096958572a6b3adbfcb20cf46"} err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal_openshift-machine-config-operator_24e19692c959ce3358ef1453cec7edef_4 in pod sandbox e74f5a2fd08893956e8bae18610eae46c6b3bd027a404da15eb7d34d125c55c1 from index: no such id: 'f227eae44fb7abb0d51682b3ca9423f8b89afb8096958572a6b3adbfcb20cf46'"
Sep 10 13:22:11.132142 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:11.132110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" event={"ID":"24e19692c959ce3358ef1453cec7edef","Type":"ContainerStarted","Data":"1d82d33710af32ec756de371b2b9eb2f9ff8ce66d7920873650d59405c17f5bc"}
Sep 10 13:22:19.005237 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:19.005191    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-llj8p"
Sep 10 13:22:19.043701 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:22:19.043649    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-19.ec2.internal" podStartSLOduration=102.043632129 podStartE2EDuration="1m42.043632129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:22:11.249663733 +0000 UTC m=+236.266495704" watchObservedRunningTime="2026-09-10 13:22:19.043632129 +0000 UTC m=+244.060464117"
Sep 10 13:23:15.312859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:23:15.312828    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 13:26:43.579935 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.579887    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"]
Sep 10 13:26:43.583741 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.583716    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.590277 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.589929    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 13:26:43.590277 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.590044    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m6d87\""
Sep 10 13:26:43.590277 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.589942    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 13:26:43.590477 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.590329    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 13:26:43.594101 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.594082    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 13:26:43.614783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.614753    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"]
Sep 10 13:26:43.641681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.641647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jn2\" (UniqueName: \"kubernetes.io/projected/17e155fc-d390-49c3-9741-bc94d0b88df7-kube-api-access-m2jn2\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.641681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.641688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17e155fc-d390-49c3-9741-bc94d0b88df7-manifests\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.641907 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.641708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17e155fc-d390-49c3-9741-bc94d0b88df7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.641907 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.641786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17e155fc-d390-49c3-9741-bc94d0b88df7-tmp\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.641907 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.641832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17e155fc-d390-49c3-9741-bc94d0b88df7-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.742835 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.742794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jn2\" (UniqueName: \"kubernetes.io/projected/17e155fc-d390-49c3-9741-bc94d0b88df7-kube-api-access-m2jn2\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.743053 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.742843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17e155fc-d390-49c3-9741-bc94d0b88df7-manifests\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.743053 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.742869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17e155fc-d390-49c3-9741-bc94d0b88df7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.743053 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.742907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17e155fc-d390-49c3-9741-bc94d0b88df7-tmp\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.743053 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.742953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17e155fc-d390-49c3-9741-bc94d0b88df7-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.743339 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.743315    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/17e155fc-d390-49c3-9741-bc94d0b88df7-manifests\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.745396 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.745372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17e155fc-d390-49c3-9741-bc94d0b88df7-tmp\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.745529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.745459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/17e155fc-d390-49c3-9741-bc94d0b88df7-webhook-cert\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.745529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.745495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/17e155fc-d390-49c3-9741-bc94d0b88df7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.754361 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.754327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jn2\" (UniqueName: \"kubernetes.io/projected/17e155fc-d390-49c3-9741-bc94d0b88df7-kube-api-access-m2jn2\") pod \"opendatahub-operator-controller-manager-84d444f55-2wmkd\" (UID: \"17e155fc-d390-49c3-9741-bc94d0b88df7\") " pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:43.893512 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:43.893427    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:44.178836 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:44.178813    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"]
Sep 10 13:26:44.180531 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:26:44.180505    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e155fc_d390_49c3_9741_bc94d0b88df7.slice/crio-14ba52d7b387230be0ffdf7b9d12566eca116880e0c25e8bfc6665f366f6522e WatchSource:0}: Error finding container 14ba52d7b387230be0ffdf7b9d12566eca116880e0c25e8bfc6665f366f6522e: Status 404 returned error can't find the container with id 14ba52d7b387230be0ffdf7b9d12566eca116880e0c25e8bfc6665f366f6522e
Sep 10 13:26:44.182300 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:44.182282    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:26:45.155604 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:45.155558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd" event={"ID":"17e155fc-d390-49c3-9741-bc94d0b88df7","Type":"ContainerStarted","Data":"14ba52d7b387230be0ffdf7b9d12566eca116880e0c25e8bfc6665f366f6522e"}
Sep 10 13:26:48.174378 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:48.174341    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd" event={"ID":"17e155fc-d390-49c3-9741-bc94d0b88df7","Type":"ContainerStarted","Data":"8ea58c2d980e84909f60ea7e639cc1e6c27f878941ba77cc9c601b389a316751"}
Sep 10 13:26:49.180060 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:49.180027    2578 generic.go:356] "Generic (PLEG): container finished" podID="17e155fc-d390-49c3-9741-bc94d0b88df7" containerID="8ea58c2d980e84909f60ea7e639cc1e6c27f878941ba77cc9c601b389a316751" exitCode=0
Sep 10 13:26:49.180484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:49.180068    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd" event={"ID":"17e155fc-d390-49c3-9741-bc94d0b88df7","Type":"ContainerDied","Data":"8ea58c2d980e84909f60ea7e639cc1e6c27f878941ba77cc9c601b389a316751"}
Sep 10 13:26:50.186406 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:50.186376    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd" event={"ID":"17e155fc-d390-49c3-9741-bc94d0b88df7","Type":"ContainerStarted","Data":"118d2d9cb7c3eece1251bb80b8b14a821b9659a958ac83a605d071047da4cdcb"}
Sep 10 13:26:50.186801 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:50.186511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:26:50.211764 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:26:50.211708    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd" podStartSLOduration=2.878519671 podStartE2EDuration="7.211690804s" totalImagesPullingTime="3.324157774s" totalInitContainerRuntime="1.009013359s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:26:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:26:50.209994088 +0000 UTC m=+515.226826059" watchObservedRunningTime="2026-09-10 13:26:50.211690804 +0000 UTC m=+515.228522776"
Sep 10 13:27:01.192696 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:01.192668    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84d444f55-2wmkd"
Sep 10 13:27:03.612449 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.612420    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"]
Sep 10 13:27:03.623068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.623041    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.635400 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.635376    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 13:27:03.635400 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.635394    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 13:27:03.635577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.635381    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 13:27:03.651268 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.651236    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 13:27:03.652898 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.652871    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"]
Sep 10 13:27:03.709537 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.709507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phqvl\" (UniqueName: \"kubernetes.io/projected/7143a86a-20f1-44c9-83be-528e37661a57-kube-api-access-phqvl\") pod \"kube-auth-proxy-cb8b5959f-85mgm\" (UID: \"7143a86a-20f1-44c9-83be-528e37661a57\") " pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.709721 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.709543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7143a86a-20f1-44c9-83be-528e37661a57-tls-certs\") pod \"kube-auth-proxy-cb8b5959f-85mgm\" (UID: \"7143a86a-20f1-44c9-83be-528e37661a57\") " pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.709721 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.709575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7143a86a-20f1-44c9-83be-528e37661a57-tmp\") pod \"kube-auth-proxy-cb8b5959f-85mgm\" (UID: \"7143a86a-20f1-44c9-83be-528e37661a57\") " pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.810810 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.810774    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7143a86a-20f1-44c9-83be-528e37661a57-tmp\") pod \"kube-auth-proxy-cb8b5959f-85mgm\" (UID: \"7143a86a-20f1-44c9-83be-528e37661a57\") " pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.811006 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.810858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phqvl\" (UniqueName: \"kubernetes.io/projected/7143a86a-20f1-44c9-83be-528e37661a57-kube-api-access-phqvl\") pod \"kube-auth-proxy-cb8b5959f-85mgm\" (UID: \"7143a86a-20f1-44c9-83be-528e37661a57\") " pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.811006 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.810888    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7143a86a-20f1-44c9-83be-528e37661a57-tls-certs\") pod \"kube-auth-proxy-cb8b5959f-85mgm\" (UID: \"7143a86a-20f1-44c9-83be-528e37661a57\") " pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.813441 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.813417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7143a86a-20f1-44c9-83be-528e37661a57-tmp\") pod \"kube-auth-proxy-cb8b5959f-85mgm\" (UID: \"7143a86a-20f1-44c9-83be-528e37661a57\") " pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.813686 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.813666    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7143a86a-20f1-44c9-83be-528e37661a57-tls-certs\") pod \"kube-auth-proxy-cb8b5959f-85mgm\" (UID: \"7143a86a-20f1-44c9-83be-528e37661a57\") " pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.825721 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.825694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phqvl\" (UniqueName: \"kubernetes.io/projected/7143a86a-20f1-44c9-83be-528e37661a57-kube-api-access-phqvl\") pod \"kube-auth-proxy-cb8b5959f-85mgm\" (UID: \"7143a86a-20f1-44c9-83be-528e37661a57\") " pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:03.932362 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:03.932257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"
Sep 10 13:27:04.233498 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:04.233465    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm"]
Sep 10 13:27:04.238557 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:27:04.238534    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7143a86a_20f1_44c9_83be_528e37661a57.slice/crio-61b1dc00063664a984fcfd4d8bd04ac6f33a54b65e73339dd614742f08bed4a7 WatchSource:0}: Error finding container 61b1dc00063664a984fcfd4d8bd04ac6f33a54b65e73339dd614742f08bed4a7: Status 404 returned error can't find the container with id 61b1dc00063664a984fcfd4d8bd04ac6f33a54b65e73339dd614742f08bed4a7
Sep 10 13:27:05.211486 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.211454    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"]
Sep 10 13:27:05.214414 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.214062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.219702 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.219674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 13:27:05.219850 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.219770    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4tc5x\""
Sep 10 13:27:05.220024 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.219970    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 13:27:05.220303 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.220008    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 13:27:05.220499 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.220442    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 13:27:05.238985 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.238919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"]
Sep 10 13:27:05.254108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.254062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm" event={"ID":"7143a86a-20f1-44c9-83be-528e37661a57","Type":"ContainerStarted","Data":"61b1dc00063664a984fcfd4d8bd04ac6f33a54b65e73339dd614742f08bed4a7"}
Sep 10 13:27:05.327285 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.327163    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.327285 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.327263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.327285 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.327289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmgb4\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-kube-api-access-qmgb4\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.327678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.327352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.327678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.327395    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.327678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.327450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0ff2c9-f513-4f49-bccb-914842d95923-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.327678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.327516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.428804 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.428762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.428804 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.428812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmgb4\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-kube-api-access-qmgb4\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.429056 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.428843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.429056 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.428868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.429056 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.428904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0ff2c9-f513-4f49-bccb-914842d95923-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.429056 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.428950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.429056 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.428984    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.430356 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.430327    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.432171 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.432145    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.432569 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.432546    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.432717 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.432687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0ff2c9-f513-4f49-bccb-914842d95923-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.432840 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.432781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.438903 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.438859    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.439765 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.439737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmgb4\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-kube-api-access-qmgb4\") pod \"istiod-openshift-gateway-7b7995d5cd-gqmbh\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:05.528259 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:05.528223    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:07.185465 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:07.185387    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"]
Sep 10 13:27:07.188566 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:27:07.188542    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0ff2c9_f513_4f49_bccb_914842d95923.slice/crio-bd1d9bf67b8a6dcba246bc20692ff921a5f0a5138f3617b873fb34c3722edfa2 WatchSource:0}: Error finding container bd1d9bf67b8a6dcba246bc20692ff921a5f0a5138f3617b873fb34c3722edfa2: Status 404 returned error can't find the container with id bd1d9bf67b8a6dcba246bc20692ff921a5f0a5138f3617b873fb34c3722edfa2
Sep 10 13:27:07.264671 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:07.264635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh" event={"ID":"1a0ff2c9-f513-4f49-bccb-914842d95923","Type":"ContainerStarted","Data":"bd1d9bf67b8a6dcba246bc20692ff921a5f0a5138f3617b873fb34c3722edfa2"}
Sep 10 13:27:07.266085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:07.266052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm" event={"ID":"7143a86a-20f1-44c9-83be-528e37661a57","Type":"ContainerStarted","Data":"142bbc4202ccca320cdf03227f73717c1abd2e4e33e3105b3611413099d3c098"}
Sep 10 13:27:11.299976 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:11.299933    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 13:27:11.300249 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:11.300019    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 13:27:12.287405 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:12.287370    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh" event={"ID":"1a0ff2c9-f513-4f49-bccb-914842d95923","Type":"ContainerStarted","Data":"21b2ae59234b91ce4c572823924f46c3ef729eb46f5b46e101b3cda4b6830106"}
Sep 10 13:27:12.287589 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:12.287526    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:12.289243 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:12.289219    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-gqmbh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 13:27:12.289351 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:12.289266    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh" podUID="1a0ff2c9-f513-4f49-bccb-914842d95923" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 13:27:12.318327 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:12.318277    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cb8b5959f-85mgm" podStartSLOduration=6.620257194 podStartE2EDuration="9.318259855s" totalImagesPullingTime="2.698002661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:27:07.284235957 +0000 UTC m=+532.301067926" watchObservedRunningTime="2026-09-10 13:27:12.318259855 +0000 UTC m=+537.335091827"
Sep 10 13:27:12.318683 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:12.318359    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh" podStartSLOduration=3.209332607 podStartE2EDuration="7.31835461s" totalImagesPullingTime="4.109022003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:27:12.316733994 +0000 UTC m=+537.333565969" watchObservedRunningTime="2026-09-10 13:27:12.31835461 +0000 UTC m=+537.335186582"
Sep 10 13:27:13.293122 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:13.293058    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:15.335237 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:15.335181    2578 scope.go:117] "RemoveContainer" containerID="71dd60db23c8b16a2ec598579d58d0eb54ed22d0801c0cdc6af6f8c35a2a174a"
Sep 10 13:27:15.344130 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:15.344111    2578 scope.go:117] "RemoveContainer" containerID="868a7e497645addcf3242388da3d73d72d52ecdc91d430180dea346b36e19560"
Sep 10 13:27:15.351078 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:15.351058    2578 scope.go:117] "RemoveContainer" containerID="cc9e25e03b3acf344a04ee189b365df80d880b931c5fa8cd91f521d68b313ca4"
Sep 10 13:27:15.357924 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:15.357902    2578 scope.go:117] "RemoveContainer" containerID="8f6c108a8b5edd917f2d17d79e322e27f757a5930c109c7bfa846f7ca4aa58e4"
Sep 10 13:27:15.365308 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:15.365290    2578 scope.go:117] "RemoveContainer" containerID="572d6d9de4af11c14805ac81427f1c550251369dcfc99db9de6b28540a512712"
Sep 10 13:27:15.372419 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:15.372397    2578 scope.go:117] "RemoveContainer" containerID="6110c70a99725c1e23433940b58c0ea2e522893318c5d3a955e08c1ee0a997ad"
Sep 10 13:27:15.379617 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:15.379588    2578 scope.go:117] "RemoveContainer" containerID="fb573ad4781cfb98bba8433ee6e360fe168114e72960727aa8479572dd249539"
Sep 10 13:27:15.386024 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:15.385999    2578 scope.go:117] "RemoveContainer" containerID="ae96eda1c32c82e3103e0ce9b97953f530cf6e556d2b140c9b5a8cadef0030dc"
Sep 10 13:27:25.683685 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.683649    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"]
Sep 10 13:27:25.688683 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.688664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:27:25.692299 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.692273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19ac0b83-57a5-4e97-b2f0-6f5535a27b80-cert\") pod \"kserve-controller-manager-d6f4b65b8-4x8vz\" (UID: \"19ac0b83-57a5-4e97-b2f0-6f5535a27b80\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:27:25.692412 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.692314    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76vb\" (UniqueName: \"kubernetes.io/projected/19ac0b83-57a5-4e97-b2f0-6f5535a27b80-kube-api-access-f76vb\") pod \"kserve-controller-manager-d6f4b65b8-4x8vz\" (UID: \"19ac0b83-57a5-4e97-b2f0-6f5535a27b80\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:27:25.694152 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.694132    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 13:27:25.699027 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.699004    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lskmd\""
Sep 10 13:27:25.761366 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.761332    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"]
Sep 10 13:27:25.764435 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.764413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:27:25.772008 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.771988    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 13:27:25.774690 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.774674    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jgwkq\""
Sep 10 13:27:25.787132 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.787102    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"]
Sep 10 13:27:25.793043 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.793010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19ac0b83-57a5-4e97-b2f0-6f5535a27b80-cert\") pod \"kserve-controller-manager-d6f4b65b8-4x8vz\" (UID: \"19ac0b83-57a5-4e97-b2f0-6f5535a27b80\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:27:25.793221 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.793068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76vb\" (UniqueName: \"kubernetes.io/projected/19ac0b83-57a5-4e97-b2f0-6f5535a27b80-kube-api-access-f76vb\") pod \"kserve-controller-manager-d6f4b65b8-4x8vz\" (UID: \"19ac0b83-57a5-4e97-b2f0-6f5535a27b80\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:27:25.795628 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.795582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19ac0b83-57a5-4e97-b2f0-6f5535a27b80-cert\") pod \"kserve-controller-manager-d6f4b65b8-4x8vz\" (UID: \"19ac0b83-57a5-4e97-b2f0-6f5535a27b80\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:27:25.815923 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.815892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76vb\" (UniqueName: \"kubernetes.io/projected/19ac0b83-57a5-4e97-b2f0-6f5535a27b80-kube-api-access-f76vb\") pod \"kserve-controller-manager-d6f4b65b8-4x8vz\" (UID: \"19ac0b83-57a5-4e97-b2f0-6f5535a27b80\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:27:25.817454 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.817427    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"]
Sep 10 13:27:25.893863 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.893828    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j57xq\" (UniqueName: \"kubernetes.io/projected/235f19de-7f94-40ef-9e2e-d4edd3ed63b6-kube-api-access-j57xq\") pod \"llmisvc-controller-manager-777c6bd7d5-q2hl2\" (UID: \"235f19de-7f94-40ef-9e2e-d4edd3ed63b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:27:25.894040 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.893894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/235f19de-7f94-40ef-9e2e-d4edd3ed63b6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q2hl2\" (UID: \"235f19de-7f94-40ef-9e2e-d4edd3ed63b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:27:25.994852 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.994811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j57xq\" (UniqueName: \"kubernetes.io/projected/235f19de-7f94-40ef-9e2e-d4edd3ed63b6-kube-api-access-j57xq\") pod \"llmisvc-controller-manager-777c6bd7d5-q2hl2\" (UID: \"235f19de-7f94-40ef-9e2e-d4edd3ed63b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:27:25.995029 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.994879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/235f19de-7f94-40ef-9e2e-d4edd3ed63b6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q2hl2\" (UID: \"235f19de-7f94-40ef-9e2e-d4edd3ed63b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:27:25.997590 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:25.997560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/235f19de-7f94-40ef-9e2e-d4edd3ed63b6-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q2hl2\" (UID: \"235f19de-7f94-40ef-9e2e-d4edd3ed63b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:27:26.000087 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:26.000067    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:27:26.014482 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:26.014453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j57xq\" (UniqueName: \"kubernetes.io/projected/235f19de-7f94-40ef-9e2e-d4edd3ed63b6-kube-api-access-j57xq\") pod \"llmisvc-controller-manager-777c6bd7d5-q2hl2\" (UID: \"235f19de-7f94-40ef-9e2e-d4edd3ed63b6\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:27:26.073608 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:26.073576    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:27:26.284473 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:26.284387    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"]
Sep 10 13:27:26.287935 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:27:26.287908    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ac0b83_57a5_4e97_b2f0_6f5535a27b80.slice/crio-244456b2e66ce3c59ef1d2fdaf0d79ca649726071b5696bc86432568db301cfd WatchSource:0}: Error finding container 244456b2e66ce3c59ef1d2fdaf0d79ca649726071b5696bc86432568db301cfd: Status 404 returned error can't find the container with id 244456b2e66ce3c59ef1d2fdaf0d79ca649726071b5696bc86432568db301cfd
Sep 10 13:27:26.344255 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:26.344221    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz" event={"ID":"19ac0b83-57a5-4e97-b2f0-6f5535a27b80","Type":"ContainerStarted","Data":"244456b2e66ce3c59ef1d2fdaf0d79ca649726071b5696bc86432568db301cfd"}
Sep 10 13:27:26.357586 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:26.357557    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"]
Sep 10 13:27:26.362160 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:27:26.362135    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod235f19de_7f94_40ef_9e2e_d4edd3ed63b6.slice/crio-70f4fe511c7ede04d2b164d42fc3530fcd067bb87724aeb162f9660ae5f1898c WatchSource:0}: Error finding container 70f4fe511c7ede04d2b164d42fc3530fcd067bb87724aeb162f9660ae5f1898c: Status 404 returned error can't find the container with id 70f4fe511c7ede04d2b164d42fc3530fcd067bb87724aeb162f9660ae5f1898c
Sep 10 13:27:27.354847 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:27.354811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2" event={"ID":"235f19de-7f94-40ef-9e2e-d4edd3ed63b6","Type":"ContainerStarted","Data":"70f4fe511c7ede04d2b164d42fc3530fcd067bb87724aeb162f9660ae5f1898c"}
Sep 10 13:27:27.767666 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:27.767626    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"]
Sep 10 13:27:27.767867 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:27.767837    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh" podUID="1a0ff2c9-f513-4f49-bccb-914842d95923" containerName="discovery" containerID="cri-o://21b2ae59234b91ce4c572823924f46c3ef729eb46f5b46e101b3cda4b6830106" gracePeriod=30
Sep 10 13:27:28.368797 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.368759    2578 generic.go:356] "Generic (PLEG): container finished" podID="1a0ff2c9-f513-4f49-bccb-914842d95923" containerID="21b2ae59234b91ce4c572823924f46c3ef729eb46f5b46e101b3cda4b6830106" exitCode=0
Sep 10 13:27:28.369241 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.368847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh" event={"ID":"1a0ff2c9-f513-4f49-bccb-914842d95923","Type":"ContainerDied","Data":"21b2ae59234b91ce4c572823924f46c3ef729eb46f5b46e101b3cda4b6830106"}
Sep 10 13:27:28.521397 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.521371    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:28.615158 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.615135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-ca-configmap\") pod \"1a0ff2c9-f513-4f49-bccb-914842d95923\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") "
Sep 10 13:27:28.615362 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.615173    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-kubeconfig\") pod \"1a0ff2c9-f513-4f49-bccb-914842d95923\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") "
Sep 10 13:27:28.615362 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.615224    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-dns-cert\") pod \"1a0ff2c9-f513-4f49-bccb-914842d95923\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") "
Sep 10 13:27:28.615362 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.615245    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-cacerts\") pod \"1a0ff2c9-f513-4f49-bccb-914842d95923\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") "
Sep 10 13:27:28.615362 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.615272    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-token\") pod \"1a0ff2c9-f513-4f49-bccb-914842d95923\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") "
Sep 10 13:27:28.615362 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.615295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0ff2c9-f513-4f49-bccb-914842d95923-local-certs\") pod \"1a0ff2c9-f513-4f49-bccb-914842d95923\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") "
Sep 10 13:27:28.615362 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.615331    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmgb4\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-kube-api-access-qmgb4\") pod \"1a0ff2c9-f513-4f49-bccb-914842d95923\" (UID: \"1a0ff2c9-f513-4f49-bccb-914842d95923\") "
Sep 10 13:27:28.615887 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.615846    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1a0ff2c9-f513-4f49-bccb-914842d95923" (UID: "1a0ff2c9-f513-4f49-bccb-914842d95923"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:27:28.618274 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.618132    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-cacerts" (OuterVolumeSpecName: "cacerts") pod "1a0ff2c9-f513-4f49-bccb-914842d95923" (UID: "1a0ff2c9-f513-4f49-bccb-914842d95923"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:27:28.618274 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.618215    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-kube-api-access-qmgb4" (OuterVolumeSpecName: "kube-api-access-qmgb4") pod "1a0ff2c9-f513-4f49-bccb-914842d95923" (UID: "1a0ff2c9-f513-4f49-bccb-914842d95923"). InnerVolumeSpecName "kube-api-access-qmgb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:27:28.618274 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.618221    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0ff2c9-f513-4f49-bccb-914842d95923-local-certs" (OuterVolumeSpecName: "local-certs") pod "1a0ff2c9-f513-4f49-bccb-914842d95923" (UID: "1a0ff2c9-f513-4f49-bccb-914842d95923"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:27:28.618274 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.618263    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a0ff2c9-f513-4f49-bccb-914842d95923" (UID: "1a0ff2c9-f513-4f49-bccb-914842d95923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:27:28.618484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.618308    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1a0ff2c9-f513-4f49-bccb-914842d95923" (UID: "1a0ff2c9-f513-4f49-bccb-914842d95923"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:27:28.618484 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.618325    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1a0ff2c9-f513-4f49-bccb-914842d95923" (UID: "1a0ff2c9-f513-4f49-bccb-914842d95923"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:27:28.716272 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.716236    2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-kubeconfig\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:27:28.716272 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.716271    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-dns-cert\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:27:28.716496 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.716286    2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a0ff2c9-f513-4f49-bccb-914842d95923-cacerts\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:27:28.716496 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.716300    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-token\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:27:28.716496 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.716314    2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0ff2c9-f513-4f49-bccb-914842d95923-local-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:27:28.716496 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.716327    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmgb4\" (UniqueName: \"kubernetes.io/projected/1a0ff2c9-f513-4f49-bccb-914842d95923-kube-api-access-qmgb4\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:27:28.716496 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:28.716340    2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a0ff2c9-f513-4f49-bccb-914842d95923-istio-csr-ca-configmap\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:27:29.374143 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:29.374111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz" event={"ID":"19ac0b83-57a5-4e97-b2f0-6f5535a27b80","Type":"ContainerStarted","Data":"d54536c706161cc5d01fe869a87a168ab3b594d58b29430cbb21ce12537ede0b"}
Sep 10 13:27:29.374547 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:29.374242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:27:29.375478 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:29.375460    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh" event={"ID":"1a0ff2c9-f513-4f49-bccb-914842d95923","Type":"ContainerDied","Data":"bd1d9bf67b8a6dcba246bc20692ff921a5f0a5138f3617b873fb34c3722edfa2"}
Sep 10 13:27:29.375572 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:29.375484    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"
Sep 10 13:27:29.375572 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:29.375501    2578 scope.go:117] "RemoveContainer" containerID="21b2ae59234b91ce4c572823924f46c3ef729eb46f5b46e101b3cda4b6830106"
Sep 10 13:27:29.407162 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:29.407112    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz" podStartSLOduration=2.144599138 podStartE2EDuration="4.407095236s" totalImagesPullingTime="2.262496098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:27:29.406051748 +0000 UTC m=+554.422883719" watchObservedRunningTime="2026-09-10 13:27:29.407095236 +0000 UTC m=+554.423927207"
Sep 10 13:27:29.437809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:29.437773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"]
Sep 10 13:27:29.441092 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:29.441058    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh"]
Sep 10 13:27:29.451903 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:29.451870    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a0ff2c9-f513-4f49-bccb-914842d95923" path="/var/lib/kubelet/pods/1a0ff2c9-f513-4f49-bccb-914842d95923/volumes"
Sep 10 13:27:30.381101 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:30.381062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2" event={"ID":"235f19de-7f94-40ef-9e2e-d4edd3ed63b6","Type":"ContainerStarted","Data":"e47cea0c8bf5ce5c914e59f1784bcb6ab2045b7a3bc6e16d4a7ea8ff43aa8a32"}
Sep 10 13:27:30.381613 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:30.381130    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:27:30.416704 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:30.416650    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2" podStartSLOduration=2.453049996 podStartE2EDuration="5.416633793s" totalImagesPullingTime="2.963583797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:27:30.416256448 +0000 UTC m=+555.433088419" watchObservedRunningTime="2026-09-10 13:27:30.416633793 +0000 UTC m=+555.433465764"
Sep 10 13:27:33.293547 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:33.293489    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-gqmbh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.19:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 13:27:33.293991 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:27:33.293579    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gqmbh" podUID="1a0ff2c9-f513-4f49-bccb-914842d95923" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.19:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 13:28:00.387526 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:00.387492    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-4x8vz"
Sep 10 13:28:01.388930 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:01.388901    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q2hl2"
Sep 10 13:28:34.825858 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.825823    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-j7drr"]
Sep 10 13:28:34.826377 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.826114    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a0ff2c9-f513-4f49-bccb-914842d95923" containerName="discovery"
Sep 10 13:28:34.826377 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.826125    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a0ff2c9-f513-4f49-bccb-914842d95923" containerName="discovery"
Sep 10 13:28:34.826377 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.826186    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a0ff2c9-f513-4f49-bccb-914842d95923" containerName="discovery"
Sep 10 13:28:34.828095 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.828079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-j7drr"
Sep 10 13:28:34.831964 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.831944    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 13:28:34.832424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.832412    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 13:28:34.832878 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.832862    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-hkbfc\""
Sep 10 13:28:34.851232 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.851187    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-j7drr"]
Sep 10 13:28:34.941517 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:34.941479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wld2h\" (UniqueName: \"kubernetes.io/projected/6c1a6add-5be2-448d-8b0f-7b8823bba5b9-kube-api-access-wld2h\") pod \"authorino-operator-76b478bd78-j7drr\" (UID: \"6c1a6add-5be2-448d-8b0f-7b8823bba5b9\") " pod="kuadrant-system/authorino-operator-76b478bd78-j7drr"
Sep 10 13:28:35.042056 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:35.042014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wld2h\" (UniqueName: \"kubernetes.io/projected/6c1a6add-5be2-448d-8b0f-7b8823bba5b9-kube-api-access-wld2h\") pod \"authorino-operator-76b478bd78-j7drr\" (UID: \"6c1a6add-5be2-448d-8b0f-7b8823bba5b9\") " pod="kuadrant-system/authorino-operator-76b478bd78-j7drr"
Sep 10 13:28:35.055763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:35.055732    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wld2h\" (UniqueName: \"kubernetes.io/projected/6c1a6add-5be2-448d-8b0f-7b8823bba5b9-kube-api-access-wld2h\") pod \"authorino-operator-76b478bd78-j7drr\" (UID: \"6c1a6add-5be2-448d-8b0f-7b8823bba5b9\") " pod="kuadrant-system/authorino-operator-76b478bd78-j7drr"
Sep 10 13:28:35.137674 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:35.137579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-j7drr"
Sep 10 13:28:35.420897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:35.420872    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-j7drr"]
Sep 10 13:28:35.423304 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:28:35.423273    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1a6add_5be2_448d_8b0f_7b8823bba5b9.slice/crio-2d14e545702a986a3fa88cbb371fead8e1d410538352b681d6e65d376e6172bb WatchSource:0}: Error finding container 2d14e545702a986a3fa88cbb371fead8e1d410538352b681d6e65d376e6172bb: Status 404 returned error can't find the container with id 2d14e545702a986a3fa88cbb371fead8e1d410538352b681d6e65d376e6172bb
Sep 10 13:28:35.658572 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:35.658540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-j7drr" event={"ID":"6c1a6add-5be2-448d-8b0f-7b8823bba5b9","Type":"ContainerStarted","Data":"2d14e545702a986a3fa88cbb371fead8e1d410538352b681d6e65d376e6172bb"}
Sep 10 13:28:36.979486 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:36.979451    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"]
Sep 10 13:28:36.985506 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:36.985477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:28:36.988976 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:36.988948    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6wp5g\""
Sep 10 13:28:37.003887 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:37.003829    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"]
Sep 10 13:28:37.055080 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:37.055042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzrw6\" (UniqueName: \"kubernetes.io/projected/575a054f-0aaf-45c2-b8e9-ce2d29b7f198-kube-api-access-dzrw6\") pod \"limitador-operator-controller-manager-74495c69f8-hz46n\" (UID: \"575a054f-0aaf-45c2-b8e9-ce2d29b7f198\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:28:37.155696 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:37.155647    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzrw6\" (UniqueName: \"kubernetes.io/projected/575a054f-0aaf-45c2-b8e9-ce2d29b7f198-kube-api-access-dzrw6\") pod \"limitador-operator-controller-manager-74495c69f8-hz46n\" (UID: \"575a054f-0aaf-45c2-b8e9-ce2d29b7f198\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:28:37.173765 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:37.173729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzrw6\" (UniqueName: \"kubernetes.io/projected/575a054f-0aaf-45c2-b8e9-ce2d29b7f198-kube-api-access-dzrw6\") pod \"limitador-operator-controller-manager-74495c69f8-hz46n\" (UID: \"575a054f-0aaf-45c2-b8e9-ce2d29b7f198\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:28:37.298085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:37.297990    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:28:37.958861 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:37.958780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"]
Sep 10 13:28:37.962554 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:28:37.962513    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575a054f_0aaf_45c2_b8e9_ce2d29b7f198.slice/crio-72a82ee8730444e4efb2e931d2c1ee02fd12ad6c9836ad866341b346674b8cd3 WatchSource:0}: Error finding container 72a82ee8730444e4efb2e931d2c1ee02fd12ad6c9836ad866341b346674b8cd3: Status 404 returned error can't find the container with id 72a82ee8730444e4efb2e931d2c1ee02fd12ad6c9836ad866341b346674b8cd3
Sep 10 13:28:38.675579 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:38.675524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n" event={"ID":"575a054f-0aaf-45c2-b8e9-ce2d29b7f198","Type":"ContainerStarted","Data":"72a82ee8730444e4efb2e931d2c1ee02fd12ad6c9836ad866341b346674b8cd3"}
Sep 10 13:28:38.677153 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:38.677124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-j7drr" event={"ID":"6c1a6add-5be2-448d-8b0f-7b8823bba5b9","Type":"ContainerStarted","Data":"3a22c9a89898d8392ab2cacfae1acb4f0320a93642f88c744429148c2000386a"}
Sep 10 13:28:38.677284 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:38.677247    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-j7drr"
Sep 10 13:28:38.722644 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:38.722590    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-j7drr" podStartSLOduration=2.43012616 podStartE2EDuration="4.722571023s" totalImagesPullingTime="2.292444863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:28:38.721332607 +0000 UTC m=+623.738164577" watchObservedRunningTime="2026-09-10 13:28:38.722571023 +0000 UTC m=+623.739403066"
Sep 10 13:28:39.164885 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.164811    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"]
Sep 10 13:28:39.167472 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.167450    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"
Sep 10 13:28:39.170094 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.170020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bktjc\" (UniqueName: \"kubernetes.io/projected/f0bbb328-8ba6-4850-a81c-983074807cc6-kube-api-access-bktjc\") pod \"dns-operator-controller-manager-7b84cd7c9c-qcnjp\" (UID: \"f0bbb328-8ba6-4850-a81c-983074807cc6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"
Sep 10 13:28:39.170740 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.170553    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 13:28:39.170740 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.170621    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-852fz\""
Sep 10 13:28:39.186280 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.186248    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"]
Sep 10 13:28:39.270773 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.270739    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bktjc\" (UniqueName: \"kubernetes.io/projected/f0bbb328-8ba6-4850-a81c-983074807cc6-kube-api-access-bktjc\") pod \"dns-operator-controller-manager-7b84cd7c9c-qcnjp\" (UID: \"f0bbb328-8ba6-4850-a81c-983074807cc6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"
Sep 10 13:28:39.287039 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.287004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bktjc\" (UniqueName: \"kubernetes.io/projected/f0bbb328-8ba6-4850-a81c-983074807cc6-kube-api-access-bktjc\") pod \"dns-operator-controller-manager-7b84cd7c9c-qcnjp\" (UID: \"f0bbb328-8ba6-4850-a81c-983074807cc6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"
Sep 10 13:28:39.489930 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.479581    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"
Sep 10 13:28:39.684893 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.684858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n" event={"ID":"575a054f-0aaf-45c2-b8e9-ce2d29b7f198","Type":"ContainerStarted","Data":"f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72"}
Sep 10 13:28:39.769028 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.768966    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n" podStartSLOduration=2.274781821 podStartE2EDuration="3.768946045s" totalImagesPullingTime="1.494164224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:28:39.70791188 +0000 UTC m=+624.724743866" watchObservedRunningTime="2026-09-10 13:28:39.768946045 +0000 UTC m=+624.785778073"
Sep 10 13:28:39.770335 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:39.770311    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"]
Sep 10 13:28:39.773831 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:28:39.773801    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0bbb328_8ba6_4850_a81c_983074807cc6.slice/crio-f11ddcd816a23e70792db1bfaae1f378a28204bc4c2d138ab086f89fa8ee9272 WatchSource:0}: Error finding container f11ddcd816a23e70792db1bfaae1f378a28204bc4c2d138ab086f89fa8ee9272: Status 404 returned error can't find the container with id f11ddcd816a23e70792db1bfaae1f378a28204bc4c2d138ab086f89fa8ee9272
Sep 10 13:28:40.690674 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:40.690635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp" event={"ID":"f0bbb328-8ba6-4850-a81c-983074807cc6","Type":"ContainerStarted","Data":"f11ddcd816a23e70792db1bfaae1f378a28204bc4c2d138ab086f89fa8ee9272"}
Sep 10 13:28:40.691119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:40.690790    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:28:43.706104 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:43.706071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp" event={"ID":"f0bbb328-8ba6-4850-a81c-983074807cc6","Type":"ContainerStarted","Data":"8bdbb5eaf2364797434e8cd8dbd8e334d314c6e03d24cc376704d81b8bb398a0"}
Sep 10 13:28:43.706592 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:43.706215    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"
Sep 10 13:28:43.734147 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:43.734093    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp" podStartSLOduration=1.470333615 podStartE2EDuration="4.734076623s" totalImagesPullingTime="3.263743008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:28:43.733421064 +0000 UTC m=+628.750253035" watchObservedRunningTime="2026-09-10 13:28:43.734076623 +0000 UTC m=+628.750908620"
Sep 10 13:28:43.904873 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:43.904835    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"]
Sep 10 13:28:43.907222 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:43.907182    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:28:43.910026 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:43.910004    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6svwj\""
Sep 10 13:28:43.930475 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:43.930440    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"]
Sep 10 13:28:44.005871 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:44.005838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6377cfb-0103-46b1-96f6-8af86cd82aaa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" (UID: \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:28:44.006040 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:44.005885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95844\" (UniqueName: \"kubernetes.io/projected/d6377cfb-0103-46b1-96f6-8af86cd82aaa-kube-api-access-95844\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" (UID: \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:28:44.106620 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:44.106583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95844\" (UniqueName: \"kubernetes.io/projected/d6377cfb-0103-46b1-96f6-8af86cd82aaa-kube-api-access-95844\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" (UID: \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:28:44.106802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:44.106664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6377cfb-0103-46b1-96f6-8af86cd82aaa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" (UID: \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:28:44.107013 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:44.106998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6377cfb-0103-46b1-96f6-8af86cd82aaa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" (UID: \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:28:44.116686 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:44.116658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95844\" (UniqueName: \"kubernetes.io/projected/d6377cfb-0103-46b1-96f6-8af86cd82aaa-kube-api-access-95844\") pod \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" (UID: \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:28:44.216860 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:44.216816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:28:44.495554 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:44.495506    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"]
Sep 10 13:28:44.497466 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:28:44.497440    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6377cfb_0103_46b1_96f6_8af86cd82aaa.slice/crio-f426c87eca484e6b371c6cf19755a874a0aabf29b9176ed45b42f74c243151db WatchSource:0}: Error finding container f426c87eca484e6b371c6cf19755a874a0aabf29b9176ed45b42f74c243151db: Status 404 returned error can't find the container with id f426c87eca484e6b371c6cf19755a874a0aabf29b9176ed45b42f74c243151db
Sep 10 13:28:44.711989 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:44.711895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52" event={"ID":"d6377cfb-0103-46b1-96f6-8af86cd82aaa","Type":"ContainerStarted","Data":"f426c87eca484e6b371c6cf19755a874a0aabf29b9176ed45b42f74c243151db"}
Sep 10 13:28:48.732426 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:48.732394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52" event={"ID":"d6377cfb-0103-46b1-96f6-8af86cd82aaa","Type":"ContainerStarted","Data":"dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77"}
Sep 10 13:28:48.732814 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:48.732526    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:28:48.782085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:48.782028    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52" podStartSLOduration=2.381677677 podStartE2EDuration="5.782013994s" totalImagesPullingTime="3.400336317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:28:48.781661789 +0000 UTC m=+633.798493760" watchObservedRunningTime="2026-09-10 13:28:48.782013994 +0000 UTC m=+633.798845964"
Sep 10 13:28:49.687940 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:49.687899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-j7drr"
Sep 10 13:28:50.928235 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:50.928186    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"]
Sep 10 13:28:50.930904 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:50.930880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:50.946937 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:50.946912    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 13:28:50.947152 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:50.947139    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 13:28:50.947670 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:50.947654    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kpd5p\""
Sep 10 13:28:50.953068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:50.953048    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"]
Sep 10 13:28:50.960436 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:50.960410    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlwn\" (UniqueName: \"kubernetes.io/projected/9dea981c-a6d9-42ed-af66-599fed284de7-kube-api-access-khlwn\") pod \"kuadrant-console-plugin-fbdd4f445-xhdvm\" (UID: \"9dea981c-a6d9-42ed-af66-599fed284de7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:50.960544 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:50.960459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9dea981c-a6d9-42ed-af66-599fed284de7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xhdvm\" (UID: \"9dea981c-a6d9-42ed-af66-599fed284de7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:50.960589 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:50.960561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dea981c-a6d9-42ed-af66-599fed284de7-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xhdvm\" (UID: \"9dea981c-a6d9-42ed-af66-599fed284de7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:51.061859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.061822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlwn\" (UniqueName: \"kubernetes.io/projected/9dea981c-a6d9-42ed-af66-599fed284de7-kube-api-access-khlwn\") pod \"kuadrant-console-plugin-fbdd4f445-xhdvm\" (UID: \"9dea981c-a6d9-42ed-af66-599fed284de7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:51.061859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.061871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9dea981c-a6d9-42ed-af66-599fed284de7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xhdvm\" (UID: \"9dea981c-a6d9-42ed-af66-599fed284de7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:51.062129 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.061930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dea981c-a6d9-42ed-af66-599fed284de7-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xhdvm\" (UID: \"9dea981c-a6d9-42ed-af66-599fed284de7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:51.062645 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.062619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9dea981c-a6d9-42ed-af66-599fed284de7-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xhdvm\" (UID: \"9dea981c-a6d9-42ed-af66-599fed284de7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:51.064327 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.064307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9dea981c-a6d9-42ed-af66-599fed284de7-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xhdvm\" (UID: \"9dea981c-a6d9-42ed-af66-599fed284de7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:51.080086 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.080050    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlwn\" (UniqueName: \"kubernetes.io/projected/9dea981c-a6d9-42ed-af66-599fed284de7-kube-api-access-khlwn\") pod \"kuadrant-console-plugin-fbdd4f445-xhdvm\" (UID: \"9dea981c-a6d9-42ed-af66-599fed284de7\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:51.239301 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.239257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"
Sep 10 13:28:51.582548 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.582463    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm"]
Sep 10 13:28:51.585399 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:28:51.585370    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dea981c_a6d9_42ed_af66_599fed284de7.slice/crio-c7970820d4f8c134a18b677a4bdc14fd4f6e29c7e80ba6c67e88a4190376dcb1 WatchSource:0}: Error finding container c7970820d4f8c134a18b677a4bdc14fd4f6e29c7e80ba6c67e88a4190376dcb1: Status 404 returned error can't find the container with id c7970820d4f8c134a18b677a4bdc14fd4f6e29c7e80ba6c67e88a4190376dcb1
Sep 10 13:28:51.697860 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.697828    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:28:51.747559 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:51.747494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm" event={"ID":"9dea981c-a6d9-42ed-af66-599fed284de7","Type":"ContainerStarted","Data":"c7970820d4f8c134a18b677a4bdc14fd4f6e29c7e80ba6c67e88a4190376dcb1"}
Sep 10 13:28:54.715302 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:54.715273    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-qcnjp"
Sep 10 13:28:56.773530 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:56.773495    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm" event={"ID":"9dea981c-a6d9-42ed-af66-599fed284de7","Type":"ContainerStarted","Data":"97087742c9ee7f107c9000f637332973b653cc6fa0ce9ea54bddc3190d193be4"}
Sep 10 13:28:56.800916 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:56.800861    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xhdvm" podStartSLOduration=2.624582036 podStartE2EDuration="6.800845229s" totalImagesPullingTime="4.176263193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:28:56.800013112 +0000 UTC m=+641.816845083" watchObservedRunningTime="2026-09-10 13:28:56.800845229 +0000 UTC m=+641.817677233"
Sep 10 13:28:59.739347 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:28:59.739315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:29:02.775025 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.774988    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"]
Sep 10 13:29:02.775454 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.775183    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" containerName="manager" containerID="cri-o://dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77" gracePeriod=2
Sep 10 13:29:02.799034 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.799008    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"]
Sep 10 13:29:02.846046 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.846011    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"]
Sep 10 13:29:02.846248 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.846227    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n" podUID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" containerName="manager" containerID="cri-o://f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72" gracePeriod=2
Sep 10 13:29:02.849598 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:02.849551    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:29:02.852261 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.851804    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"]
Sep 10 13:29:02.877938 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.877909    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"]
Sep 10 13:29:02.878341 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.878321    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" containerName="manager"
Sep 10 13:29:02.878341 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.878340    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" containerName="manager"
Sep 10 13:29:02.878511 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.878351    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" containerName="manager"
Sep 10 13:29:02.878511 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.878357    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" containerName="manager"
Sep 10 13:29:02.878511 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.878431    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" containerName="manager"
Sep 10 13:29:02.878511 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.878441    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" containerName="manager"
Sep 10 13:29:02.880546 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.880524    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:02.892785 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:02.892748    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:29:02.896790 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:02.896760    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hz46n\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:29:02.929796 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.929769    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"]
Sep 10 13:29:02.944118 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.944086    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"]
Sep 10 13:29:02.946139 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.946123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"
Sep 10 13:29:02.967950 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.967922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcv4\" (UniqueName: \"kubernetes.io/projected/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-kube-api-access-xfcv4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hlbl9\" (UID: \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:02.968079 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.967982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hlbl9\" (UID: \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:02.973188 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:02.973158    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"]
Sep 10 13:29:03.038153 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:03.038117    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:29:03.043789 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:03.043758    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hz46n\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:29:03.069224 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.069162    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcv4\" (UniqueName: \"kubernetes.io/projected/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-kube-api-access-xfcv4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hlbl9\" (UID: \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:03.069395 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.069255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hlbl9\" (UID: \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:03.069395 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.069287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgwp\" (UniqueName: \"kubernetes.io/projected/2677b59e-dacf-4ed4-a4dd-36ac8095e2ee-kube-api-access-ddgwp\") pod \"limitador-operator-controller-manager-74495c69f8-psqmm\" (UID: \"2677b59e-dacf-4ed4-a4dd-36ac8095e2ee\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"
Sep 10 13:29:03.069687 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.069665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hlbl9\" (UID: \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:03.082245 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.082223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcv4\" (UniqueName: \"kubernetes.io/projected/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-kube-api-access-xfcv4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-hlbl9\" (UID: \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:03.094739 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.094716    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:29:03.097357 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.097341    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:29:03.101123 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:03.101101    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:29:03.104285 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:03.104253    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hz46n\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:29:03.106527 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:03.106506    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-sxj52\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:29:03.109590 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:03.109572    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-hz46n\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:29:03.170057 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.170018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6377cfb-0103-46b1-96f6-8af86cd82aaa-extensions-socket-volume\") pod \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\" (UID: \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\") "
Sep 10 13:29:03.170371 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.170085    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzrw6\" (UniqueName: \"kubernetes.io/projected/575a054f-0aaf-45c2-b8e9-ce2d29b7f198-kube-api-access-dzrw6\") pod \"575a054f-0aaf-45c2-b8e9-ce2d29b7f198\" (UID: \"575a054f-0aaf-45c2-b8e9-ce2d29b7f198\") "
Sep 10 13:29:03.170371 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.170135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95844\" (UniqueName: \"kubernetes.io/projected/d6377cfb-0103-46b1-96f6-8af86cd82aaa-kube-api-access-95844\") pod \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\" (UID: \"d6377cfb-0103-46b1-96f6-8af86cd82aaa\") "
Sep 10 13:29:03.170371 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.170302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgwp\" (UniqueName: \"kubernetes.io/projected/2677b59e-dacf-4ed4-a4dd-36ac8095e2ee-kube-api-access-ddgwp\") pod \"limitador-operator-controller-manager-74495c69f8-psqmm\" (UID: \"2677b59e-dacf-4ed4-a4dd-36ac8095e2ee\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"
Sep 10 13:29:03.170711 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.170686    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6377cfb-0103-46b1-96f6-8af86cd82aaa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d6377cfb-0103-46b1-96f6-8af86cd82aaa" (UID: "d6377cfb-0103-46b1-96f6-8af86cd82aaa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:29:03.172337 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.172314    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575a054f-0aaf-45c2-b8e9-ce2d29b7f198-kube-api-access-dzrw6" (OuterVolumeSpecName: "kube-api-access-dzrw6") pod "575a054f-0aaf-45c2-b8e9-ce2d29b7f198" (UID: "575a054f-0aaf-45c2-b8e9-ce2d29b7f198"). InnerVolumeSpecName "kube-api-access-dzrw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:29:03.172422 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.172401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6377cfb-0103-46b1-96f6-8af86cd82aaa-kube-api-access-95844" (OuterVolumeSpecName: "kube-api-access-95844") pod "d6377cfb-0103-46b1-96f6-8af86cd82aaa" (UID: "d6377cfb-0103-46b1-96f6-8af86cd82aaa"). InnerVolumeSpecName "kube-api-access-95844". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:29:03.181350 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.181329    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgwp\" (UniqueName: \"kubernetes.io/projected/2677b59e-dacf-4ed4-a4dd-36ac8095e2ee-kube-api-access-ddgwp\") pod \"limitador-operator-controller-manager-74495c69f8-psqmm\" (UID: \"2677b59e-dacf-4ed4-a4dd-36ac8095e2ee\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"
Sep 10 13:29:03.232220 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.232152    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:03.256775 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.256744    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"
Sep 10 13:29:03.271669 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.271643    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95844\" (UniqueName: \"kubernetes.io/projected/d6377cfb-0103-46b1-96f6-8af86cd82aaa-kube-api-access-95844\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:29:03.271669 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.271668    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6377cfb-0103-46b1-96f6-8af86cd82aaa-extensions-socket-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:29:03.271811 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.271679    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzrw6\" (UniqueName: \"kubernetes.io/projected/575a054f-0aaf-45c2-b8e9-ce2d29b7f198-kube-api-access-dzrw6\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:29:03.451295 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.451263    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" path="/var/lib/kubelet/pods/575a054f-0aaf-45c2-b8e9-ce2d29b7f198/volumes"
Sep 10 13:29:03.451667 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.451653    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" path="/var/lib/kubelet/pods/d6377cfb-0103-46b1-96f6-8af86cd82aaa/volumes"
Sep 10 13:29:03.569858 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:29:03.569828    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3d349a_329b_4dfc_b1b1_6d892756e6f6.slice/crio-63b021d2507929370102d4c865fa138368dbb91616a28c764245154021489355 WatchSource:0}: Error finding container 63b021d2507929370102d4c865fa138368dbb91616a28c764245154021489355: Status 404 returned error can't find the container with id 63b021d2507929370102d4c865fa138368dbb91616a28c764245154021489355
Sep 10 13:29:03.577386 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.577362    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"]
Sep 10 13:29:03.600622 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.600594    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"]
Sep 10 13:29:03.607249 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:29:03.607217    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2677b59e_dacf_4ed4_a4dd_36ac8095e2ee.slice/crio-8f6587530fd1b41c534161f703662a76ec7124bd289e603e036639275771fdd9 WatchSource:0}: Error finding container 8f6587530fd1b41c534161f703662a76ec7124bd289e603e036639275771fdd9: Status 404 returned error can't find the container with id 8f6587530fd1b41c534161f703662a76ec7124bd289e603e036639275771fdd9
Sep 10 13:29:03.806396 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.806345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm" event={"ID":"2677b59e-dacf-4ed4-a4dd-36ac8095e2ee","Type":"ContainerStarted","Data":"ad51a3bd2793101c0ba8c6afd36c73be17355104033ac3985050a3ecd522981e"}
Sep 10 13:29:03.806807 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.806414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm" event={"ID":"2677b59e-dacf-4ed4-a4dd-36ac8095e2ee","Type":"ContainerStarted","Data":"8f6587530fd1b41c534161f703662a76ec7124bd289e603e036639275771fdd9"}
Sep 10 13:29:03.806807 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.806482    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"
Sep 10 13:29:03.808258 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.808222    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6377cfb-0103-46b1-96f6-8af86cd82aaa" containerID="dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77" exitCode=0
Sep 10 13:29:03.808381 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.808311    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-sxj52"
Sep 10 13:29:03.808381 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.808366    2578 scope.go:117] "RemoveContainer" containerID="dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77"
Sep 10 13:29:03.810899 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.810866    2578 generic.go:356] "Generic (PLEG): container finished" podID="575a054f-0aaf-45c2-b8e9-ce2d29b7f198" containerID="f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72" exitCode=0
Sep 10 13:29:03.811038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.811018    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-hz46n"
Sep 10 13:29:03.813551 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.813524    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9" event={"ID":"ed3d349a-329b-4dfc-b1b1-6d892756e6f6","Type":"ContainerStarted","Data":"8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec"}
Sep 10 13:29:03.813649 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.813568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9" event={"ID":"ed3d349a-329b-4dfc-b1b1-6d892756e6f6","Type":"ContainerStarted","Data":"63b021d2507929370102d4c865fa138368dbb91616a28c764245154021489355"}
Sep 10 13:29:03.817932 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.817911    2578 scope.go:117] "RemoveContainer" containerID="dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77"
Sep 10 13:29:03.818227 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:03.818161    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77\": container with ID starting with dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77 not found: ID does not exist" containerID="dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77"
Sep 10 13:29:03.818304 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.818227    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77"} err="failed to get container status \"dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77\": rpc error: code = NotFound desc = could not find container \"dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77\": container with ID starting with dc79b197ba2da9252c3cdd683a8c6c4f33e70d68b20908969fbd2ee7e50ffc77 not found: ID does not exist"
Sep 10 13:29:03.818304 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.818252    2578 scope.go:117] "RemoveContainer" containerID="f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72"
Sep 10 13:29:03.825747 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.825731    2578 scope.go:117] "RemoveContainer" containerID="f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72"
Sep 10 13:29:03.826013 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:03.825994    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72\": container with ID starting with f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72 not found: ID does not exist" containerID="f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72"
Sep 10 13:29:03.826068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.826021    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72"} err="failed to get container status \"f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72\": rpc error: code = NotFound desc = could not find container \"f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72\": container with ID starting with f83239509e074f0175ee00c6eda5fd01f80517b186e1beb1b301b969f1544a72 not found: ID does not exist"
Sep 10 13:29:03.903924 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:03.903870    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm" podStartSLOduration=1.9038521720000001 podStartE2EDuration="1.903852172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:03.842182124 +0000 UTC m=+648.859014095" watchObservedRunningTime="2026-09-10 13:29:03.903852172 +0000 UTC m=+648.920684127"
Sep 10 13:29:04.819731 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:04.819702    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:06.404334 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.404285    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9" podStartSLOduration=4.404269551 podStartE2EDuration="4.404269551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:03.916885722 +0000 UTC m=+648.933717694" watchObservedRunningTime="2026-09-10 13:29:06.404269551 +0000 UTC m=+651.421101522"
Sep 10 13:29:06.405673 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.405653    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"]
Sep 10 13:29:06.408537 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.408522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:06.442783 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.442748    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"]
Sep 10 13:29:06.497811 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.497779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9708ec74-1e0e-4253-9566-48e2ba024d5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5hp96\" (UID: \"9708ec74-1e0e-4253-9566-48e2ba024d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:06.497986 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.497859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jspw2\" (UniqueName: \"kubernetes.io/projected/9708ec74-1e0e-4253-9566-48e2ba024d5b-kube-api-access-jspw2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5hp96\" (UID: \"9708ec74-1e0e-4253-9566-48e2ba024d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:06.599272 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.599236    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jspw2\" (UniqueName: \"kubernetes.io/projected/9708ec74-1e0e-4253-9566-48e2ba024d5b-kube-api-access-jspw2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5hp96\" (UID: \"9708ec74-1e0e-4253-9566-48e2ba024d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:06.599488 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.599288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9708ec74-1e0e-4253-9566-48e2ba024d5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5hp96\" (UID: \"9708ec74-1e0e-4253-9566-48e2ba024d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:06.599698 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.599678    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9708ec74-1e0e-4253-9566-48e2ba024d5b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5hp96\" (UID: \"9708ec74-1e0e-4253-9566-48e2ba024d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:06.613929 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.613899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jspw2\" (UniqueName: \"kubernetes.io/projected/9708ec74-1e0e-4253-9566-48e2ba024d5b-kube-api-access-jspw2\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5hp96\" (UID: \"9708ec74-1e0e-4253-9566-48e2ba024d5b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:06.718745 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:06.718644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:07.017380 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:07.017351    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"]
Sep 10 13:29:07.020773 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:29:07.020744    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9708ec74_1e0e_4253_9566_48e2ba024d5b.slice/crio-126efb475468e97a14a74e7f3486d7398e9ffafca8343f4602fb8f16c9a07976 WatchSource:0}: Error finding container 126efb475468e97a14a74e7f3486d7398e9ffafca8343f4602fb8f16c9a07976: Status 404 returned error can't find the container with id 126efb475468e97a14a74e7f3486d7398e9ffafca8343f4602fb8f16c9a07976
Sep 10 13:29:07.836438 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:07.836401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96" event={"ID":"9708ec74-1e0e-4253-9566-48e2ba024d5b","Type":"ContainerStarted","Data":"9944758f1a1be65440ab5d2dae3aa7af3cfeb85d6f44738b0112e2cdc7b14386"}
Sep 10 13:29:07.836848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:07.836452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96" event={"ID":"9708ec74-1e0e-4253-9566-48e2ba024d5b","Type":"ContainerStarted","Data":"126efb475468e97a14a74e7f3486d7398e9ffafca8343f4602fb8f16c9a07976"}
Sep 10 13:29:07.836848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:07.836507    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:07.953556 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:07.953503    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96" podStartSLOduration=1.953488019 podStartE2EDuration="1.953488019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:07.951017913 +0000 UTC m=+652.967849883" watchObservedRunningTime="2026-09-10 13:29:07.953488019 +0000 UTC m=+652.970319989"
Sep 10 13:29:14.822155 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:14.822127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-psqmm"
Sep 10 13:29:15.827944 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:15.827911    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:18.844306 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:18.844276    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:29:18.922824 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:18.922794    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"]
Sep 10 13:29:18.923000 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:18.922974    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9" podUID="ed3d349a-329b-4dfc-b1b1-6d892756e6f6" containerName="manager" containerID="cri-o://8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec" gracePeriod=10
Sep 10 13:29:19.217285 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.217252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:19.312351 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.312318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfcv4\" (UniqueName: \"kubernetes.io/projected/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-kube-api-access-xfcv4\") pod \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\" (UID: \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\") "
Sep 10 13:29:19.312537 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.312411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-extensions-socket-volume\") pod \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\" (UID: \"ed3d349a-329b-4dfc-b1b1-6d892756e6f6\") "
Sep 10 13:29:19.312812 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.312786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ed3d349a-329b-4dfc-b1b1-6d892756e6f6" (UID: "ed3d349a-329b-4dfc-b1b1-6d892756e6f6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:29:19.314298 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.314280    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-kube-api-access-xfcv4" (OuterVolumeSpecName: "kube-api-access-xfcv4") pod "ed3d349a-329b-4dfc-b1b1-6d892756e6f6" (UID: "ed3d349a-329b-4dfc-b1b1-6d892756e6f6"). InnerVolumeSpecName "kube-api-access-xfcv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:29:19.412987 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.412894    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcv4\" (UniqueName: \"kubernetes.io/projected/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-kube-api-access-xfcv4\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:29:19.412987 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.412927    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed3d349a-329b-4dfc-b1b1-6d892756e6f6-extensions-socket-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:29:19.892292 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.892253    2578 generic.go:356] "Generic (PLEG): container finished" podID="ed3d349a-329b-4dfc-b1b1-6d892756e6f6" containerID="8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec" exitCode=0
Sep 10 13:29:19.892766 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.892309    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"
Sep 10 13:29:19.892766 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.892335    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9" event={"ID":"ed3d349a-329b-4dfc-b1b1-6d892756e6f6","Type":"ContainerDied","Data":"8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec"}
Sep 10 13:29:19.892766 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.892441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9" event={"ID":"ed3d349a-329b-4dfc-b1b1-6d892756e6f6","Type":"ContainerDied","Data":"63b021d2507929370102d4c865fa138368dbb91616a28c764245154021489355"}
Sep 10 13:29:19.892766 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.892472    2578 scope.go:117] "RemoveContainer" containerID="8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec"
Sep 10 13:29:19.900801 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.900782    2578 scope.go:117] "RemoveContainer" containerID="8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec"
Sep 10 13:29:19.901069 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:19.901051    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec\": container with ID starting with 8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec not found: ID does not exist" containerID="8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec"
Sep 10 13:29:19.901145 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.901076    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec"} err="failed to get container status \"8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec\": rpc error: code = NotFound desc = could not find container \"8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec\": container with ID starting with 8bd12c4a970485bec444df53e5b897aa866a4350de7ac75241482a9d1e2f38ec not found: ID does not exist"
Sep 10 13:29:19.921859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.921831    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"]
Sep 10 13:29:19.933022 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:19.932992    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-hlbl9"]
Sep 10 13:29:21.454135 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:21.454054    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed3d349a-329b-4dfc-b1b1-6d892756e6f6" path="/var/lib/kubelet/pods/ed3d349a-329b-4dfc-b1b1-6d892756e6f6/volumes"
Sep 10 13:29:40.256625 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.256586    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lkxrm"]
Sep 10 13:29:40.257040 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.256917    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3d349a-329b-4dfc-b1b1-6d892756e6f6" containerName="manager"
Sep 10 13:29:40.257040 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.256928    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3d349a-329b-4dfc-b1b1-6d892756e6f6" containerName="manager"
Sep 10 13:29:40.257040 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.256982    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3d349a-329b-4dfc-b1b1-6d892756e6f6" containerName="manager"
Sep 10 13:29:40.259795 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.259772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:40.262953 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.262932    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 13:29:40.275561 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.275533    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lkxrm"]
Sep 10 13:29:40.349782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.349750    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lkxrm"]
Sep 10 13:29:40.395329 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.395296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bsvg\" (UniqueName: \"kubernetes.io/projected/183e4c9b-3b83-46f3-8562-69d4b8909c2d-kube-api-access-5bsvg\") pod \"limitador-limitador-8588749977-lkxrm\" (UID: \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\") " pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:40.395329 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.395333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/183e4c9b-3b83-46f3-8562-69d4b8909c2d-config-file\") pod \"limitador-limitador-8588749977-lkxrm\" (UID: \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\") " pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:40.496826 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.496789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bsvg\" (UniqueName: \"kubernetes.io/projected/183e4c9b-3b83-46f3-8562-69d4b8909c2d-kube-api-access-5bsvg\") pod \"limitador-limitador-8588749977-lkxrm\" (UID: \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\") " pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:40.496826 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.496827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/183e4c9b-3b83-46f3-8562-69d4b8909c2d-config-file\") pod \"limitador-limitador-8588749977-lkxrm\" (UID: \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\") " pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:40.497466 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.497446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/183e4c9b-3b83-46f3-8562-69d4b8909c2d-config-file\") pod \"limitador-limitador-8588749977-lkxrm\" (UID: \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\") " pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:40.507610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.507548    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bsvg\" (UniqueName: \"kubernetes.io/projected/183e4c9b-3b83-46f3-8562-69d4b8909c2d-kube-api-access-5bsvg\") pod \"limitador-limitador-8588749977-lkxrm\" (UID: \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\") " pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:40.569910 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.569869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:40.845378 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.845348    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lkxrm"]
Sep 10 13:29:40.847818 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:29:40.847790    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183e4c9b_3b83_46f3_8562_69d4b8909c2d.slice/crio-e3fc8a98105b2a334a7d267c97349058f75026c9605c0449e0fae928338526f8 WatchSource:0}: Error finding container e3fc8a98105b2a334a7d267c97349058f75026c9605c0449e0fae928338526f8: Status 404 returned error can't find the container with id e3fc8a98105b2a334a7d267c97349058f75026c9605c0449e0fae928338526f8
Sep 10 13:29:40.988654 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:40.988617    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm" event={"ID":"183e4c9b-3b83-46f3-8562-69d4b8909c2d","Type":"ContainerStarted","Data":"e3fc8a98105b2a334a7d267c97349058f75026c9605c0449e0fae928338526f8"}
Sep 10 13:29:42.999321 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:42.999284    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm" event={"ID":"183e4c9b-3b83-46f3-8562-69d4b8909c2d","Type":"ContainerStarted","Data":"112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54"}
Sep 10 13:29:42.999803 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:42.999348    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:43.021376 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:43.021323    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm" podStartSLOduration=1.676099668 podStartE2EDuration="3.021306188s" totalImagesPullingTime="1.34520652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:29:43.019060666 +0000 UTC m=+688.035892638" watchObservedRunningTime="2026-09-10 13:29:43.021306188 +0000 UTC m=+688.038138160"
Sep 10 13:29:54.005155 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:54.005123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:56.217693 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:56.217658    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lkxrm"]
Sep 10 13:29:56.218088 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:56.217891    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm" podUID="183e4c9b-3b83-46f3-8562-69d4b8909c2d" containerName="limitador" containerID="cri-o://112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54" gracePeriod=30
Sep 10 13:29:56.770244 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:56.770219    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:56.811465 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:56.811390    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bsvg\" (UniqueName: \"kubernetes.io/projected/183e4c9b-3b83-46f3-8562-69d4b8909c2d-kube-api-access-5bsvg\") pod \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\" (UID: \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\") "
Sep 10 13:29:56.811465 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:56.811434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/183e4c9b-3b83-46f3-8562-69d4b8909c2d-config-file\") pod \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\" (UID: \"183e4c9b-3b83-46f3-8562-69d4b8909c2d\") "
Sep 10 13:29:56.811854 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:56.811825    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/183e4c9b-3b83-46f3-8562-69d4b8909c2d-config-file" (OuterVolumeSpecName: "config-file") pod "183e4c9b-3b83-46f3-8562-69d4b8909c2d" (UID: "183e4c9b-3b83-46f3-8562-69d4b8909c2d"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:29:56.813488 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:56.813462    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183e4c9b-3b83-46f3-8562-69d4b8909c2d-kube-api-access-5bsvg" (OuterVolumeSpecName: "kube-api-access-5bsvg") pod "183e4c9b-3b83-46f3-8562-69d4b8909c2d" (UID: "183e4c9b-3b83-46f3-8562-69d4b8909c2d"). InnerVolumeSpecName "kube-api-access-5bsvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:29:56.912374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:56.912336    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bsvg\" (UniqueName: \"kubernetes.io/projected/183e4c9b-3b83-46f3-8562-69d4b8909c2d-kube-api-access-5bsvg\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:29:56.912374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:56.912365    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/183e4c9b-3b83-46f3-8562-69d4b8909c2d-config-file\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:29:57.069949 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.069852    2578 generic.go:356] "Generic (PLEG): container finished" podID="183e4c9b-3b83-46f3-8562-69d4b8909c2d" containerID="112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54" exitCode=0
Sep 10 13:29:57.069949 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.069909    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm"
Sep 10 13:29:57.069949 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.069935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm" event={"ID":"183e4c9b-3b83-46f3-8562-69d4b8909c2d","Type":"ContainerDied","Data":"112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54"}
Sep 10 13:29:57.070249 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.070000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-lkxrm" event={"ID":"183e4c9b-3b83-46f3-8562-69d4b8909c2d","Type":"ContainerDied","Data":"e3fc8a98105b2a334a7d267c97349058f75026c9605c0449e0fae928338526f8"}
Sep 10 13:29:57.070249 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.070023    2578 scope.go:117] "RemoveContainer" containerID="112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54"
Sep 10 13:29:57.078356 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.078338    2578 scope.go:117] "RemoveContainer" containerID="112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54"
Sep 10 13:29:57.078642 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:29:57.078625    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54\": container with ID starting with 112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54 not found: ID does not exist" containerID="112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54"
Sep 10 13:29:57.078705 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.078649    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54"} err="failed to get container status \"112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54\": rpc error: code = NotFound desc = could not find container \"112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54\": container with ID starting with 112ba60028b76f8fcb98a3ce67404575e20e5dace0cb587f99cc546f9b379a54 not found: ID does not exist"
Sep 10 13:29:57.095879 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.095851    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lkxrm"]
Sep 10 13:29:57.102791 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.102769    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-lkxrm"]
Sep 10 13:29:57.451240 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:29:57.451133    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="183e4c9b-3b83-46f3-8562-69d4b8909c2d" path="/var/lib/kubelet/pods/183e4c9b-3b83-46f3-8562-69d4b8909c2d/volumes"
Sep 10 13:30:01.989695 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:01.989663    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"]
Sep 10 13:30:01.990063 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:01.990001    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="183e4c9b-3b83-46f3-8562-69d4b8909c2d" containerName="limitador"
Sep 10 13:30:01.990063 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:01.990013    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="183e4c9b-3b83-46f3-8562-69d4b8909c2d" containerName="limitador"
Sep 10 13:30:01.990063 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:01.990063    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="183e4c9b-3b83-46f3-8562-69d4b8909c2d" containerName="limitador"
Sep 10 13:30:01.994520 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:01.994500    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:02.001590 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:02.001562    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"postgres-creds\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" type="*v1.Secret"
Sep 10 13:30:02.001718 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:02.001588    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"postgres-69fb4594cf-jdcxn\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" podUID="95f8e0d2-6c06-4cb5-812d-2fd07963ad2c" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:02.001718 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:02.001601    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-ghvg7\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ghvg7\"" type="*v1.Secret"
Sep 10 13:30:02.001718 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:02.001634    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 10 13:30:02.001718 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:02.001642    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-19.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-138-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 10 13:30:02.035359 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:02.035320    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"]
Sep 10 13:30:02.053636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:02.053594    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmjqv\" (UniqueName: \"kubernetes.io/projected/95f8e0d2-6c06-4cb5-812d-2fd07963ad2c-kube-api-access-wmjqv\") pod \"postgres-69fb4594cf-jdcxn\" (UID: \"95f8e0d2-6c06-4cb5-812d-2fd07963ad2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:02.053812 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:02.053709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95f8e0d2-6c06-4cb5-812d-2fd07963ad2c-data\") pod \"postgres-69fb4594cf-jdcxn\" (UID: \"95f8e0d2-6c06-4cb5-812d-2fd07963ad2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:02.155007 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:02.154972    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95f8e0d2-6c06-4cb5-812d-2fd07963ad2c-data\") pod \"postgres-69fb4594cf-jdcxn\" (UID: \"95f8e0d2-6c06-4cb5-812d-2fd07963ad2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:02.155225 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:02.155033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjqv\" (UniqueName: \"kubernetes.io/projected/95f8e0d2-6c06-4cb5-812d-2fd07963ad2c-kube-api-access-wmjqv\") pod \"postgres-69fb4594cf-jdcxn\" (UID: \"95f8e0d2-6c06-4cb5-812d-2fd07963ad2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:02.155467 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:02.155448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/95f8e0d2-6c06-4cb5-812d-2fd07963ad2c-data\") pod \"postgres-69fb4594cf-jdcxn\" (UID: \"95f8e0d2-6c06-4cb5-812d-2fd07963ad2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:02.963127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:02.963094    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 13:30:02.970503 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:02.970467    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 13:30:03.164415 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:03.164376    2578 projected.go:291] Couldn't get configMap odh-ai-gateway-infra/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:30:03.164415 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:03.164419    2578 projected.go:196] Error preparing data for projected volume kube-api-access-wmjqv for pod odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn: failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:30:03.164840 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:03.164477    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/95f8e0d2-6c06-4cb5-812d-2fd07963ad2c-kube-api-access-wmjqv podName:95f8e0d2-6c06-4cb5-812d-2fd07963ad2c nodeName:}" failed. No retries permitted until 2026-09-10 13:30:03.664460866 +0000 UTC m=+708.681292817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wmjqv" (UniqueName: "kubernetes.io/projected/95f8e0d2-6c06-4cb5-812d-2fd07963ad2c-kube-api-access-wmjqv") pod "postgres-69fb4594cf-jdcxn" (UID: "95f8e0d2-6c06-4cb5-812d-2fd07963ad2c") : failed to sync configmap cache: timed out waiting for the condition
Sep 10 13:30:03.169819 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:03.169796    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-ghvg7\""
Sep 10 13:30:03.187469 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:03.187443    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 13:30:03.668117 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:03.668082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjqv\" (UniqueName: \"kubernetes.io/projected/95f8e0d2-6c06-4cb5-812d-2fd07963ad2c-kube-api-access-wmjqv\") pod \"postgres-69fb4594cf-jdcxn\" (UID: \"95f8e0d2-6c06-4cb5-812d-2fd07963ad2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:03.670555 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:03.670533    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmjqv\" (UniqueName: \"kubernetes.io/projected/95f8e0d2-6c06-4cb5-812d-2fd07963ad2c-kube-api-access-wmjqv\") pod \"postgres-69fb4594cf-jdcxn\" (UID: \"95f8e0d2-6c06-4cb5-812d-2fd07963ad2c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:03.803808 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:03.803772    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:04.077933 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:04.077892    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"]
Sep 10 13:30:04.083062 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:30:04.083028    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f8e0d2_6c06_4cb5_812d_2fd07963ad2c.slice/crio-a8dd6a1ccda99f560ce92f169ec27880b549aa058ef3266a6bcfda829f1c9d6f WatchSource:0}: Error finding container a8dd6a1ccda99f560ce92f169ec27880b549aa058ef3266a6bcfda829f1c9d6f: Status 404 returned error can't find the container with id a8dd6a1ccda99f560ce92f169ec27880b549aa058ef3266a6bcfda829f1c9d6f
Sep 10 13:30:04.104698 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:04.104660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn" event={"ID":"95f8e0d2-6c06-4cb5-812d-2fd07963ad2c","Type":"ContainerStarted","Data":"a8dd6a1ccda99f560ce92f169ec27880b549aa058ef3266a6bcfda829f1c9d6f"}
Sep 10 13:30:10.136143 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:10.136030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn" event={"ID":"95f8e0d2-6c06-4cb5-812d-2fd07963ad2c","Type":"ContainerStarted","Data":"284a2427f3115f1144fdc4e5b2cfd180271e6b9b95868b732b67877e702e3b87"}
Sep 10 13:30:10.136571 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:10.136171    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:10.159474 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:10.159420    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn" podStartSLOduration=3.510431156 podStartE2EDuration="9.159402784s" totalImagesPullingTime="5.648971628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:10.157828206 +0000 UTC m=+715.174660169" watchObservedRunningTime="2026-09-10 13:30:10.159402784 +0000 UTC m=+715.176234755"
Sep 10 13:30:16.170106 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:16.170076    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-jdcxn"
Sep 10 13:30:24.572097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.572063    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d84575dd-tgdjx"]
Sep 10 13:30:24.577007 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.576987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:24.581336 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.581315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 13:30:24.583032 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.583014    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 13:30:24.583331 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.583308    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vtl9v\""
Sep 10 13:30:24.585010 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.584994    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 13:30:24.590505 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.590481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d84575dd-tgdjx"]
Sep 10 13:30:24.642592 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.642561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-metrics-certs\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:24.642777 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.642677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-webhook-cert\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:24.642777 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.642705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72dhk\" (UniqueName: \"kubernetes.io/projected/208a7914-9b62-42f7-b431-067644d00bb5-kube-api-access-72dhk\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:24.743108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.743065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-metrics-certs\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:24.743304 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.743173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-webhook-cert\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:24.743304 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.743210    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72dhk\" (UniqueName: \"kubernetes.io/projected/208a7914-9b62-42f7-b431-067644d00bb5-kube-api-access-72dhk\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:24.743401 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:24.743322    2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 10 13:30:24.743442 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:24.743401    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-webhook-cert podName:208a7914-9b62-42f7-b431-067644d00bb5 nodeName:}" failed. No retries permitted until 2026-09-10 13:30:25.243379301 +0000 UTC m=+730.260211250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-webhook-cert") pod "maas-controller-68d84575dd-tgdjx" (UID: "208a7914-9b62-42f7-b431-067644d00bb5") : secret "maas-controller-webhook-cert" not found
Sep 10 13:30:24.745770 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.745742    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-metrics-certs\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:24.776342 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.776302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72dhk\" (UniqueName: \"kubernetes.io/projected/208a7914-9b62-42f7-b431-067644d00bb5-kube-api-access-72dhk\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:24.827366 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.827285    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-xpch5"]
Sep 10 13:30:24.830731 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.830713    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:24.860465 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.860434    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-xpch5"]
Sep 10 13:30:24.945160 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.945101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq29p\" (UniqueName: \"kubernetes.io/projected/f0012b8d-37b5-4964-a9cf-a845485ad693-kube-api-access-mq29p\") pod \"maas-controller-6f57b54ff6-xpch5\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") " pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:24.945160 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.945157    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-webhook-cert\") pod \"maas-controller-6f57b54ff6-xpch5\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") " pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:24.945426 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:24.945250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-metrics-certs\") pod \"maas-controller-6f57b54ff6-xpch5\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") " pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:25.046405 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.046367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-metrics-certs\") pod \"maas-controller-6f57b54ff6-xpch5\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") " pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:25.046609 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.046476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq29p\" (UniqueName: \"kubernetes.io/projected/f0012b8d-37b5-4964-a9cf-a845485ad693-kube-api-access-mq29p\") pod \"maas-controller-6f57b54ff6-xpch5\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") " pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:25.046609 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.046498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-webhook-cert\") pod \"maas-controller-6f57b54ff6-xpch5\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") " pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:25.048885 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.048858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-webhook-cert\") pod \"maas-controller-6f57b54ff6-xpch5\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") " pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:25.049019 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.048984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-metrics-certs\") pod \"maas-controller-6f57b54ff6-xpch5\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") " pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:25.058600 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.058576    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq29p\" (UniqueName: \"kubernetes.io/projected/f0012b8d-37b5-4964-a9cf-a845485ad693-kube-api-access-mq29p\") pod \"maas-controller-6f57b54ff6-xpch5\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") " pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:25.141677 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.141586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:25.248787 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.248752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-webhook-cert\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:25.251158 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.251134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-webhook-cert\") pod \"maas-controller-68d84575dd-tgdjx\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") " pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:25.417142 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.417116    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-xpch5"]
Sep 10 13:30:25.419580 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:30:25.419553    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0012b8d_37b5_4964_a9cf_a845485ad693.slice/crio-f84b39b2a97d64efef92f51ade4d3ab8c7e48a63bba324bb1e37229be580fc8b WatchSource:0}: Error finding container f84b39b2a97d64efef92f51ade4d3ab8c7e48a63bba324bb1e37229be580fc8b: Status 404 returned error can't find the container with id f84b39b2a97d64efef92f51ade4d3ab8c7e48a63bba324bb1e37229be580fc8b
Sep 10 13:30:25.487329 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.487299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:25.767212 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:25.767051    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d84575dd-tgdjx"]
Sep 10 13:30:25.769616 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:30:25.769587    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208a7914_9b62_42f7_b431_067644d00bb5.slice/crio-3668813d8ee372f4f548f6eb115b7dca339ccb7e9d3d5e14adf2b3b0e25da2c3 WatchSource:0}: Error finding container 3668813d8ee372f4f548f6eb115b7dca339ccb7e9d3d5e14adf2b3b0e25da2c3: Status 404 returned error can't find the container with id 3668813d8ee372f4f548f6eb115b7dca339ccb7e9d3d5e14adf2b3b0e25da2c3
Sep 10 13:30:26.214230 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:26.214086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-xpch5" event={"ID":"f0012b8d-37b5-4964-a9cf-a845485ad693","Type":"ContainerStarted","Data":"f84b39b2a97d64efef92f51ade4d3ab8c7e48a63bba324bb1e37229be580fc8b"}
Sep 10 13:30:26.216330 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:26.216272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d84575dd-tgdjx" event={"ID":"208a7914-9b62-42f7-b431-067644d00bb5","Type":"ContainerStarted","Data":"3668813d8ee372f4f548f6eb115b7dca339ccb7e9d3d5e14adf2b3b0e25da2c3"}
Sep 10 13:30:28.229797 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:28.229762    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d84575dd-tgdjx" event={"ID":"208a7914-9b62-42f7-b431-067644d00bb5","Type":"ContainerStarted","Data":"d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1"}
Sep 10 13:30:28.230179 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:28.229866    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:28.231601 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:28.231566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-xpch5" event={"ID":"f0012b8d-37b5-4964-a9cf-a845485ad693","Type":"ContainerStarted","Data":"fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579"}
Sep 10 13:30:28.231703 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:28.231690    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:28.258267 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:28.258215    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d84575dd-tgdjx" podStartSLOduration=1.876783275 podStartE2EDuration="4.258180411s" totalImagesPullingTime="2.381397136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:28.256844061 +0000 UTC m=+733.273676029" watchObservedRunningTime="2026-09-10 13:30:28.258180411 +0000 UTC m=+733.275012381"
Sep 10 13:30:28.291812 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:28.291754    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f57b54ff6-xpch5" podStartSLOduration=1.5669581099999998 podStartE2EDuration="4.291738415s" totalImagesPullingTime="2.724780305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:30:28.287893387 +0000 UTC m=+733.304725359" watchObservedRunningTime="2026-09-10 13:30:28.291738415 +0000 UTC m=+733.308570387"
Sep 10 13:30:39.245096 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.245064    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:30:39.245568 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.245136    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:39.352286 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.352254    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d84575dd-tgdjx"]
Sep 10 13:30:39.352460 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.352430    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68d84575dd-tgdjx" podUID="208a7914-9b62-42f7-b431-067644d00bb5" containerName="manager" containerID="cri-o://d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1" gracePeriod=30
Sep 10 13:30:39.593707 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.593676    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:39.680997 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.680962    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72dhk\" (UniqueName: \"kubernetes.io/projected/208a7914-9b62-42f7-b431-067644d00bb5-kube-api-access-72dhk\") pod \"208a7914-9b62-42f7-b431-067644d00bb5\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") "
Sep 10 13:30:39.681185 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.681029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-webhook-cert\") pod \"208a7914-9b62-42f7-b431-067644d00bb5\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") "
Sep 10 13:30:39.681185 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.681058    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-metrics-certs\") pod \"208a7914-9b62-42f7-b431-067644d00bb5\" (UID: \"208a7914-9b62-42f7-b431-067644d00bb5\") "
Sep 10 13:30:39.683172 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.683136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "208a7914-9b62-42f7-b431-067644d00bb5" (UID: "208a7914-9b62-42f7-b431-067644d00bb5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:30:39.683307 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.683188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208a7914-9b62-42f7-b431-067644d00bb5-kube-api-access-72dhk" (OuterVolumeSpecName: "kube-api-access-72dhk") pod "208a7914-9b62-42f7-b431-067644d00bb5" (UID: "208a7914-9b62-42f7-b431-067644d00bb5"). InnerVolumeSpecName "kube-api-access-72dhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:30:39.683307 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.683227    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "208a7914-9b62-42f7-b431-067644d00bb5" (UID: "208a7914-9b62-42f7-b431-067644d00bb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:30:39.781860 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.781773    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72dhk\" (UniqueName: \"kubernetes.io/projected/208a7914-9b62-42f7-b431-067644d00bb5-kube-api-access-72dhk\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:30:39.781860 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.781804    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-webhook-cert\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:30:39.781860 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:39.781813    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/208a7914-9b62-42f7-b431-067644d00bb5-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:30:40.293266 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:40.293233    2578 generic.go:356] "Generic (PLEG): container finished" podID="208a7914-9b62-42f7-b431-067644d00bb5" containerID="d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1" exitCode=0
Sep 10 13:30:40.293658 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:40.293279    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d84575dd-tgdjx"
Sep 10 13:30:40.293658 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:40.293323    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d84575dd-tgdjx" event={"ID":"208a7914-9b62-42f7-b431-067644d00bb5","Type":"ContainerDied","Data":"d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1"}
Sep 10 13:30:40.293658 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:40.293408    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d84575dd-tgdjx" event={"ID":"208a7914-9b62-42f7-b431-067644d00bb5","Type":"ContainerDied","Data":"3668813d8ee372f4f548f6eb115b7dca339ccb7e9d3d5e14adf2b3b0e25da2c3"}
Sep 10 13:30:40.293658 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:40.293436    2578 scope.go:117] "RemoveContainer" containerID="d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1"
Sep 10 13:30:40.303659 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:40.303641    2578 scope.go:117] "RemoveContainer" containerID="d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1"
Sep 10 13:30:40.303922 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:30:40.303903    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1\": container with ID starting with d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1 not found: ID does not exist" containerID="d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1"
Sep 10 13:30:40.303985 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:40.303931    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1"} err="failed to get container status \"d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1\": rpc error: code = NotFound desc = could not find container \"d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1\": container with ID starting with d12bac64849ce98457294ba59ae5d8e713c0b590e654037fd61d5014bfeffbf1 not found: ID does not exist"
Sep 10 13:30:40.320844 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:40.320817    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d84575dd-tgdjx"]
Sep 10 13:30:40.322384 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:40.322365    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68d84575dd-tgdjx"]
Sep 10 13:30:41.451479 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:30:41.451442    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="208a7914-9b62-42f7-b431-067644d00bb5" path="/var/lib/kubelet/pods/208a7914-9b62-42f7-b431-067644d00bb5/volumes"
Sep 10 13:31:01.332284 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.332250    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"]
Sep 10 13:31:01.332763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.332591    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="208a7914-9b62-42f7-b431-067644d00bb5" containerName="manager"
Sep 10 13:31:01.332763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.332601    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="208a7914-9b62-42f7-b431-067644d00bb5" containerName="manager"
Sep 10 13:31:01.332763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.332663    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="208a7914-9b62-42f7-b431-067644d00bb5" containerName="manager"
Sep 10 13:31:01.339486 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.339462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.346367 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.346018    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 13:31:01.346367 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.346339    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 13:31:01.346563 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.346549    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7mkhf\""
Sep 10 13:31:01.362543 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.362508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-metrics-certs\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.362543 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.362545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68191959-d911-4c6a-8cae-8d65cbad9d1b-openshift-service-ca\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.362707 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.362564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phjbg\" (UniqueName: \"kubernetes.io/projected/68191959-d911-4c6a-8cae-8d65cbad9d1b-kube-api-access-phjbg\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.362707 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.362584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-maas-api-tls\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.376112 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.376083    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"]
Sep 10 13:31:01.463762 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.463706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-metrics-certs\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.463762 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.463754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68191959-d911-4c6a-8cae-8d65cbad9d1b-openshift-service-ca\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.464002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.463777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phjbg\" (UniqueName: \"kubernetes.io/projected/68191959-d911-4c6a-8cae-8d65cbad9d1b-kube-api-access-phjbg\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.464002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.463811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-maas-api-tls\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.464528 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.464502    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68191959-d911-4c6a-8cae-8d65cbad9d1b-openshift-service-ca\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.466333 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.466303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-metrics-certs\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.466538 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.466359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-maas-api-tls\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.480717 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.480683    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phjbg\" (UniqueName: \"kubernetes.io/projected/68191959-d911-4c6a-8cae-8d65cbad9d1b-kube-api-access-phjbg\") pod \"maas-api-6b9dddc78b-s426b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") " pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.649440 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.649349    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:01.933081 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:01.933049    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"]
Sep 10 13:31:01.935313 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:31:01.935290    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68191959_d911_4c6a_8cae_8d65cbad9d1b.slice/crio-5bbfb67030acf771605bb96a1b4f3d77d62ef401073fc487d6d5f02e41940d48 WatchSource:0}: Error finding container 5bbfb67030acf771605bb96a1b4f3d77d62ef401073fc487d6d5f02e41940d48: Status 404 returned error can't find the container with id 5bbfb67030acf771605bb96a1b4f3d77d62ef401073fc487d6d5f02e41940d48
Sep 10 13:31:02.394687 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:02.394649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b" event={"ID":"68191959-d911-4c6a-8cae-8d65cbad9d1b","Type":"ContainerStarted","Data":"5bbfb67030acf771605bb96a1b4f3d77d62ef401073fc487d6d5f02e41940d48"}
Sep 10 13:31:02.950842 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:02.950814    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"]
Sep 10 13:31:02.953181 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:02.953165    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:02.967894 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:02.967869    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 13:31:02.968790 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:02.968759    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pcjc7\""
Sep 10 13:31:02.975492 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:02.975458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2114350c-4310-43c2-9fc9-4ef47e717a06-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-cs9hb\" (UID: \"2114350c-4310-43c2-9fc9-4ef47e717a06\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:02.975669 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:02.975552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqzsr\" (UniqueName: \"kubernetes.io/projected/2114350c-4310-43c2-9fc9-4ef47e717a06-kube-api-access-nqzsr\") pod \"payload-processing-5b54b7fbb7-cs9hb\" (UID: \"2114350c-4310-43c2-9fc9-4ef47e717a06\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:03.002164 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:03.002131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"]
Sep 10 13:31:03.076113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:03.076073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2114350c-4310-43c2-9fc9-4ef47e717a06-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-cs9hb\" (UID: \"2114350c-4310-43c2-9fc9-4ef47e717a06\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:03.076320 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:03.076131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqzsr\" (UniqueName: \"kubernetes.io/projected/2114350c-4310-43c2-9fc9-4ef47e717a06-kube-api-access-nqzsr\") pod \"payload-processing-5b54b7fbb7-cs9hb\" (UID: \"2114350c-4310-43c2-9fc9-4ef47e717a06\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:03.076945 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:03.076923    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2114350c-4310-43c2-9fc9-4ef47e717a06-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-cs9hb\" (UID: \"2114350c-4310-43c2-9fc9-4ef47e717a06\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:03.109950 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:03.109910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqzsr\" (UniqueName: \"kubernetes.io/projected/2114350c-4310-43c2-9fc9-4ef47e717a06-kube-api-access-nqzsr\") pod \"payload-processing-5b54b7fbb7-cs9hb\" (UID: \"2114350c-4310-43c2-9fc9-4ef47e717a06\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:03.263519 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:03.263475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:03.875294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:03.875269    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"]
Sep 10 13:31:03.877606 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:31:03.877573    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2114350c_4310_43c2_9fc9_4ef47e717a06.slice/crio-e1b8c38be81a185a860c4820346e17a1014385ed16027ef7e5776b9d0e04a7b4 WatchSource:0}: Error finding container e1b8c38be81a185a860c4820346e17a1014385ed16027ef7e5776b9d0e04a7b4: Status 404 returned error can't find the container with id e1b8c38be81a185a860c4820346e17a1014385ed16027ef7e5776b9d0e04a7b4
Sep 10 13:31:04.406479 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:04.406444    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb" event={"ID":"2114350c-4310-43c2-9fc9-4ef47e717a06","Type":"ContainerStarted","Data":"e1b8c38be81a185a860c4820346e17a1014385ed16027ef7e5776b9d0e04a7b4"}
Sep 10 13:31:04.407873 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:04.407849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b" event={"ID":"68191959-d911-4c6a-8cae-8d65cbad9d1b","Type":"ContainerStarted","Data":"f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315"}
Sep 10 13:31:04.408012 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:04.407979    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:08.430030 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:08.429993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb" event={"ID":"2114350c-4310-43c2-9fc9-4ef47e717a06","Type":"ContainerStarted","Data":"f12efd02d2088bf775ea0a1fd6c5e4912e2d38b86ac75038ab8fcee9ad07753e"}
Sep 10 13:31:08.430515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:08.430055    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:08.457462 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:08.457412    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb" podStartSLOduration=2.917437563 podStartE2EDuration="6.457397443s" totalImagesPullingTime="3.53995988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:08.454504825 +0000 UTC m=+773.471336796" watchObservedRunningTime="2026-09-10 13:31:08.457397443 +0000 UTC m=+773.474229414"
Sep 10 13:31:08.457828 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:08.457807    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b" podStartSLOduration=5.799386976 podStartE2EDuration="7.457799583s" totalImagesPullingTime="1.658412607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:04.471780034 +0000 UTC m=+769.488612009" watchObservedRunningTime="2026-09-10 13:31:08.457799583 +0000 UTC m=+773.474631560"
Sep 10 13:31:10.418477 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:10.418449    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:31:19.437727 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:19.437695    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-cs9hb"
Sep 10 13:31:29.116131 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.116095    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"]
Sep 10 13:31:29.118985 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.118968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.135821 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.135795    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 13:31:29.135963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.135866    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nhrvj\""
Sep 10 13:31:29.135963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.135878    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 13:31:29.135963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.135918    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 13:31:29.171983 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.171950    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"]
Sep 10 13:31:29.201636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.201603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.201636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.201647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.201913 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.201679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.201913 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.201706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b433723-19a3-4a10-b3a1-b5da19098a06-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.201913 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.201731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.201913 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.201785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b54z\" (UniqueName: \"kubernetes.io/projected/4b433723-19a3-4a10-b3a1-b5da19098a06-kube-api-access-6b54z\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.302657 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.302622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.302657 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.302663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.302897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.302681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.302897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.302793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b433723-19a3-4a10-b3a1-b5da19098a06-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.302897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.302831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.303053 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.302902    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b54z\" (UniqueName: \"kubernetes.io/projected/4b433723-19a3-4a10-b3a1-b5da19098a06-kube-api-access-6b54z\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.303105 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.303090    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.303158 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.303138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.303242 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.303164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.305264 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.305242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b433723-19a3-4a10-b3a1-b5da19098a06-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.305513 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.305491    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b433723-19a3-4a10-b3a1-b5da19098a06-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.314438 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.314417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b54z\" (UniqueName: \"kubernetes.io/projected/4b433723-19a3-4a10-b3a1-b5da19098a06-kube-api-access-6b54z\") pod \"e2e-distinct-simulated-kserve-59975d69f9-f7fmf\" (UID: \"4b433723-19a3-4a10-b3a1-b5da19098a06\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.428392 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.428299    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:29.738266 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:29.738235    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"]
Sep 10 13:31:29.741445 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:31:29.741421    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b433723_19a3_4a10_b3a1_b5da19098a06.slice/crio-da76c04b2fb9776921486929063030d56736008dc8ef536dd27caf180f03d26a WatchSource:0}: Error finding container da76c04b2fb9776921486929063030d56736008dc8ef536dd27caf180f03d26a: Status 404 returned error can't find the container with id da76c04b2fb9776921486929063030d56736008dc8ef536dd27caf180f03d26a
Sep 10 13:31:30.539188 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:30.539127    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf" event={"ID":"4b433723-19a3-4a10-b3a1-b5da19098a06","Type":"ContainerStarted","Data":"da76c04b2fb9776921486929063030d56736008dc8ef536dd27caf180f03d26a"}
Sep 10 13:31:31.544958 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:31.544919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf" event={"ID":"4b433723-19a3-4a10-b3a1-b5da19098a06","Type":"ContainerStarted","Data":"2f4a84f258e3dd29ebf535261c4d4abbe6dbc869715bd0d86d5252f0c5f34279"}
Sep 10 13:31:31.588790 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:31.588727    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf" podStartSLOduration=1.631497016 podStartE2EDuration="2.588710788s" totalImagesPullingTime="957.213772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:31:31.588399818 +0000 UTC m=+796.605231789" watchObservedRunningTime="2026-09-10 13:31:31.588710788 +0000 UTC m=+796.605542759"
Sep 10 13:31:39.429049 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:39.428952    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:39.429049 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:39.429023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:39.441491 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:39.441461    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:31:39.595689 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:31:39.595662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-f7fmf"
Sep 10 13:32:15.663364 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.663293    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"]
Sep 10 13:32:15.667141 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.667113    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.670035 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.670013    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 10 13:32:15.679606 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.679577    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"]
Sep 10 13:32:15.784097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.784060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.784097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.784100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.784329 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.784129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/654f76b4-cb45-4048-af3c-cd33c2e79fb2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.784329 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.784257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.784329 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.784299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.784433 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.784353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttqnr\" (UniqueName: \"kubernetes.io/projected/654f76b4-cb45-4048-af3c-cd33c2e79fb2-kube-api-access-ttqnr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.885355 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.885321    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.885533 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.885363    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.885533 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.885388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttqnr\" (UniqueName: \"kubernetes.io/projected/654f76b4-cb45-4048-af3c-cd33c2e79fb2-kube-api-access-ttqnr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.885533 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.885426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.885533 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.885443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.885533 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.885461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/654f76b4-cb45-4048-af3c-cd33c2e79fb2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.885855 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.885830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.885920 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.885841    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.885920 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.885899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.888150 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.888128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/654f76b4-cb45-4048-af3c-cd33c2e79fb2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.888449 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.888430    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/654f76b4-cb45-4048-af3c-cd33c2e79fb2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.899786 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.899760    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttqnr\" (UniqueName: \"kubernetes.io/projected/654f76b4-cb45-4048-af3c-cd33c2e79fb2-kube-api-access-ttqnr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5zdpt\" (UID: \"654f76b4-cb45-4048-af3c-cd33c2e79fb2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:15.979127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:15.979060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:16.260208 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:16.260167    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"]
Sep 10 13:32:16.261385 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:32:16.261356    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654f76b4_cb45_4048_af3c_cd33c2e79fb2.slice/crio-d68ae170b4fe9b88714daf2d52f5c5bfdaaee650e8d4314ee7804d3caa3241a9 WatchSource:0}: Error finding container d68ae170b4fe9b88714daf2d52f5c5bfdaaee650e8d4314ee7804d3caa3241a9: Status 404 returned error can't find the container with id d68ae170b4fe9b88714daf2d52f5c5bfdaaee650e8d4314ee7804d3caa3241a9
Sep 10 13:32:16.262825 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:16.262809    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:32:16.762073 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:16.762038    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt" event={"ID":"654f76b4-cb45-4048-af3c-cd33c2e79fb2","Type":"ContainerStarted","Data":"612595686ae6ab7515f521a81a3968715a1c4a3452b06d6b72e5246794d38247"}
Sep 10 13:32:16.762497 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:16.762086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt" event={"ID":"654f76b4-cb45-4048-af3c-cd33c2e79fb2","Type":"ContainerStarted","Data":"d68ae170b4fe9b88714daf2d52f5c5bfdaaee650e8d4314ee7804d3caa3241a9"}
Sep 10 13:32:16.786006 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:16.785862    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt" podStartSLOduration=1.581541547 podStartE2EDuration="1.78584395s" totalImagesPullingTime="204.302403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:32:16.784694218 +0000 UTC m=+841.801526195" watchObservedRunningTime="2026-09-10 13:32:16.78584395 +0000 UTC m=+841.802675920"
Sep 10 13:32:25.979886 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:25.979847    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:25.980319 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:25.979956    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:25.992250 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:25.992226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:32:26.827911 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:32:26.827879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5zdpt"
Sep 10 13:34:36.411252 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.411216    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"]
Sep 10 13:34:36.414748 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.414725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.424057 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.424031    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bc03be-serving-cert\""
Sep 10 13:34:36.432548 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.432519    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"]
Sep 10 13:34:36.529141 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.529099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-maas-api-tls\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.529141 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.529140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d81df7b-efe0-407e-b0f9-0283056e7499-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.529414 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.529225    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-metrics-certs\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.529414 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.529249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg7ps\" (UniqueName: \"kubernetes.io/projected/4d81df7b-efe0-407e-b0f9-0283056e7499-kube-api-access-kg7ps\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.630390 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.630352    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-metrics-certs\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.630390 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.630392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg7ps\" (UniqueName: \"kubernetes.io/projected/4d81df7b-efe0-407e-b0f9-0283056e7499-kube-api-access-kg7ps\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.630666 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.630429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-maas-api-tls\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.630666 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.630443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d81df7b-efe0-407e-b0f9-0283056e7499-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.631147 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.631126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d81df7b-efe0-407e-b0f9-0283056e7499-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.632908 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.632886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-metrics-certs\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.633004 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.632921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-maas-api-tls\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.649050 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.649021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg7ps\" (UniqueName: \"kubernetes.io/projected/4d81df7b-efe0-407e-b0f9-0283056e7499-kube-api-access-kg7ps\") pod \"maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:36.726508 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:36.726412    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:37.023854 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:37.023827    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"]
Sep 10 13:34:37.025161 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:34:37.025136    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d81df7b_efe0_407e_b0f9_0283056e7499.slice/crio-4e4810a01a8f4953ffc1ce1f261ba845b20e6c461b2718264161206344f40c05 WatchSource:0}: Error finding container 4e4810a01a8f4953ffc1ce1f261ba845b20e6c461b2718264161206344f40c05: Status 404 returned error can't find the container with id 4e4810a01a8f4953ffc1ce1f261ba845b20e6c461b2718264161206344f40c05
Sep 10 13:34:37.433610 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:37.433527    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj" event={"ID":"4d81df7b-efe0-407e-b0f9-0283056e7499","Type":"ContainerStarted","Data":"4e4810a01a8f4953ffc1ce1f261ba845b20e6c461b2718264161206344f40c05"}
Sep 10 13:34:38.440468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:38.440363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj" event={"ID":"4d81df7b-efe0-407e-b0f9-0283056e7499","Type":"ContainerStarted","Data":"91597e6ce29538b9cb37a0bade85fab31987faca0304445f4b5d07cb41991dc3"}
Sep 10 13:34:38.440904 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:38.440523    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:38.469183 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:38.469127    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj" podStartSLOduration=1.444280998 podStartE2EDuration="2.469109677s" totalImagesPullingTime="1.024828679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:38.466974944 +0000 UTC m=+983.483806918" watchObservedRunningTime="2026-09-10 13:34:38.469109677 +0000 UTC m=+983.485941647"
Sep 10 13:34:44.450869 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:44.450832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:34:44.960006 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:44.959976    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"]
Sep 10 13:34:44.963599 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:44.963569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:44.968165 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:44.968135    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-deba50-serving-cert\""
Sep 10 13:34:44.982107 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:44.982075    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"]
Sep 10 13:34:45.003364 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.003325    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e209893-06bb-43b0-a87e-29ffaef79ef2-openshift-service-ca\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.003564 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.003429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-metrics-certs\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.003564 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.003462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62lhc\" (UniqueName: \"kubernetes.io/projected/2e209893-06bb-43b0-a87e-29ffaef79ef2-kube-api-access-62lhc\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.003564 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.003511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-maas-api-tls\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.105077 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.105032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e209893-06bb-43b0-a87e-29ffaef79ef2-openshift-service-ca\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.105297 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.105105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-metrics-certs\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.105297 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.105132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lhc\" (UniqueName: \"kubernetes.io/projected/2e209893-06bb-43b0-a87e-29ffaef79ef2-kube-api-access-62lhc\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.105297 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.105179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-maas-api-tls\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.105483 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:34:45.105376    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-deba50-serving-cert: secret "maas-api-e2e-autores-deba50-serving-cert" not found
Sep 10 13:34:45.105483 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:34:45.105467    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-maas-api-tls podName:2e209893-06bb-43b0-a87e-29ffaef79ef2 nodeName:}" failed. No retries permitted until 2026-09-10 13:34:45.605443403 +0000 UTC m=+990.622275372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-maas-api-tls") pod "maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj" (UID: "2e209893-06bb-43b0-a87e-29ffaef79ef2") : secret "maas-api-e2e-autores-deba50-serving-cert" not found
Sep 10 13:34:45.105998 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.105969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e209893-06bb-43b0-a87e-29ffaef79ef2-openshift-service-ca\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.107960 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.107938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-metrics-certs\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.125671 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.125632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lhc\" (UniqueName: \"kubernetes.io/projected/2e209893-06bb-43b0-a87e-29ffaef79ef2-kube-api-access-62lhc\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.609880 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.609843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-maas-api-tls\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.612257 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.612234    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-maas-api-tls\") pod \"maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:45.874633 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:45.874534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:46.100108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.100070    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"]
Sep 10 13:34:46.104249 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.104225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:34:46.112852 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.112816    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-deba50\""
Sep 10 13:34:46.113713 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.113692    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm\" (UID: \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:34:46.113809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.113742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rhd\" (UniqueName: \"kubernetes.io/projected/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-kube-api-access-48rhd\") pod \"payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm\" (UID: \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:34:46.121016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.120989    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"]
Sep 10 13:34:46.122060 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.122041    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-deba50-dockercfg-x6dxj\""
Sep 10 13:34:46.170739 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.170647    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"]
Sep 10 13:34:46.173710 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:34:46.173673    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e209893_06bb_43b0_a87e_29ffaef79ef2.slice/crio-585f9f78274333c83f6878e2017f39620232530d552647b7b0ab6de7aae18812 WatchSource:0}: Error finding container 585f9f78274333c83f6878e2017f39620232530d552647b7b0ab6de7aae18812: Status 404 returned error can't find the container with id 585f9f78274333c83f6878e2017f39620232530d552647b7b0ab6de7aae18812
Sep 10 13:34:46.215019 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.214986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm\" (UID: \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:34:46.215206 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.215051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rhd\" (UniqueName: \"kubernetes.io/projected/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-kube-api-access-48rhd\") pod \"payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm\" (UID: \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:34:46.215776 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.215752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm\" (UID: \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:34:46.234054 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.234023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rhd\" (UniqueName: \"kubernetes.io/projected/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-kube-api-access-48rhd\") pod \"payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm\" (UID: \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:34:46.415012 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.414973    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:34:46.485949 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.485917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj" event={"ID":"2e209893-06bb-43b0-a87e-29ffaef79ef2","Type":"ContainerStarted","Data":"585f9f78274333c83f6878e2017f39620232530d552647b7b0ab6de7aae18812"}
Sep 10 13:34:46.490973 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.490944    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"]
Sep 10 13:34:46.494780 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.494756    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.498278 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.498258    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-3f02bf87-serving-cert\""
Sep 10 13:34:46.511678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.511303    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"]
Sep 10 13:34:46.517238 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.517211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36527a4b-aba2-4291-9f28-7ff0589aef0e-openshift-service-ca\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.517478 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.517451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbgq\" (UniqueName: \"kubernetes.io/projected/36527a4b-aba2-4291-9f28-7ff0589aef0e-kube-api-access-jkbgq\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.517602 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.517507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-maas-api-tls\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.517602 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.517529    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-metrics-certs\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.619601 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.619546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbgq\" (UniqueName: \"kubernetes.io/projected/36527a4b-aba2-4291-9f28-7ff0589aef0e-kube-api-access-jkbgq\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.620067 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.619658    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-maas-api-tls\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.620067 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.619692    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-metrics-certs\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.620067 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.619778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36527a4b-aba2-4291-9f28-7ff0589aef0e-openshift-service-ca\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.620489 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:34:46.620459    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-serving-cert: secret "maas-api-e2e-ait-3f02bf87-serving-cert" not found
Sep 10 13:34:46.620606 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:34:46.620582    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-maas-api-tls podName:36527a4b-aba2-4291-9f28-7ff0589aef0e nodeName:}" failed. No retries permitted until 2026-09-10 13:34:47.120557774 +0000 UTC m=+992.137389741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-maas-api-tls") pod "maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg" (UID: "36527a4b-aba2-4291-9f28-7ff0589aef0e") : secret "maas-api-e2e-ait-3f02bf87-serving-cert" not found
Sep 10 13:34:46.620769 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.620741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36527a4b-aba2-4291-9f28-7ff0589aef0e-openshift-service-ca\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.625297 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.623552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-metrics-certs\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.632072 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.632044    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbgq\" (UniqueName: \"kubernetes.io/projected/36527a4b-aba2-4291-9f28-7ff0589aef0e-kube-api-access-jkbgq\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:46.698167 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:46.698084    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"]
Sep 10 13:34:46.701498 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:34:46.701469    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2fc527_6f9f_4b48_94db_b2ae5008e5a2.slice/crio-3220a35f8640fbc302cd8aaf02089d461269e21e5dc1df576460c0dc4006e7ce WatchSource:0}: Error finding container 3220a35f8640fbc302cd8aaf02089d461269e21e5dc1df576460c0dc4006e7ce: Status 404 returned error can't find the container with id 3220a35f8640fbc302cd8aaf02089d461269e21e5dc1df576460c0dc4006e7ce
Sep 10 13:34:47.125042 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.125002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-maas-api-tls\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:47.127876 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.127851    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-maas-api-tls\") pod \"maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:47.409732 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.409633    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:47.494039 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.494007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj" event={"ID":"2e209893-06bb-43b0-a87e-29ffaef79ef2","Type":"ContainerStarted","Data":"ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a"}
Sep 10 13:34:47.494250 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.494133    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:47.496014 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.495948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm" event={"ID":"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2","Type":"ContainerStarted","Data":"6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5"}
Sep 10 13:34:47.496014 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.495986    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm" event={"ID":"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2","Type":"ContainerStarted","Data":"3220a35f8640fbc302cd8aaf02089d461269e21e5dc1df576460c0dc4006e7ce"}
Sep 10 13:34:47.496014 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.496015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:34:47.523892 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.523830    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj" podStartSLOduration=2.628714248 podStartE2EDuration="3.523809202s" totalImagesPullingTime="895.094954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:47.522386328 +0000 UTC m=+992.539218300" watchObservedRunningTime="2026-09-10 13:34:47.523809202 +0000 UTC m=+992.540641175"
Sep 10 13:34:47.558975 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.558926    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm" podStartSLOduration=1.558911307 podStartE2EDuration="1.558911307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:47.558486952 +0000 UTC m=+992.575318925" watchObservedRunningTime="2026-09-10 13:34:47.558911307 +0000 UTC m=+992.575743277"
Sep 10 13:34:47.705718 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:47.705632    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"]
Sep 10 13:34:47.710647 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:34:47.710616    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36527a4b_aba2_4291_9f28_7ff0589aef0e.slice/crio-5219a39157481c95f9f5cb6e52608ad2edce3221ec42a8609f40a9134f125261 WatchSource:0}: Error finding container 5219a39157481c95f9f5cb6e52608ad2edce3221ec42a8609f40a9134f125261: Status 404 returned error can't find the container with id 5219a39157481c95f9f5cb6e52608ad2edce3221ec42a8609f40a9134f125261
Sep 10 13:34:48.504933 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:48.504635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg" event={"ID":"36527a4b-aba2-4291-9f28-7ff0589aef0e","Type":"ContainerStarted","Data":"57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a"}
Sep 10 13:34:48.504933 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:48.504711    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg" event={"ID":"36527a4b-aba2-4291-9f28-7ff0589aef0e","Type":"ContainerStarted","Data":"5219a39157481c95f9f5cb6e52608ad2edce3221ec42a8609f40a9134f125261"}
Sep 10 13:34:48.529599 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:48.529547    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg" podStartSLOduration=1.953556979 podStartE2EDuration="2.529529874s" totalImagesPullingTime="575.972895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:34:48.527648283 +0000 UTC m=+993.544480244" watchObservedRunningTime="2026-09-10 13:34:48.529529874 +0000 UTC m=+993.546361843"
Sep 10 13:34:49.510807 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:49.510778    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:53.510950 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:53.510915    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:34:55.521455 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:55.521429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:34:58.507995 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:34:58.507962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:35:05.145871 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.145836    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"]
Sep 10 13:35:05.150638 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.150618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.156073 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.156046    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d6159c-serving-cert\""
Sep 10 13:35:05.187799 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.187765    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"]
Sep 10 13:35:05.295925 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.295879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.296127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.295945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-metrics-certs\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.296127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.295980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325c1350-25a8-4bc4-b791-0858b325e0c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.296127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.296061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2c66\" (UniqueName: \"kubernetes.io/projected/325c1350-25a8-4bc4-b791-0858b325e0c0-kube-api-access-p2c66\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.397634 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.397547    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.397634 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.397586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-metrics-certs\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.397634 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.397621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325c1350-25a8-4bc4-b791-0858b325e0c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.397859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.397656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2c66\" (UniqueName: \"kubernetes.io/projected/325c1350-25a8-4bc4-b791-0858b325e0c0-kube-api-access-p2c66\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.398338 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.398306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325c1350-25a8-4bc4-b791-0858b325e0c0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.400140 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.400109    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-maas-api-tls\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.400258 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.400177    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-metrics-certs\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.410331 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.410304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2c66\" (UniqueName: \"kubernetes.io/projected/325c1350-25a8-4bc4-b791-0858b325e0c0-kube-api-access-p2c66\") pod \"maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.460649 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.460615    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:05.741929 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:05.741900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"]
Sep 10 13:35:05.743605 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:35:05.743569    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325c1350_25a8_4bc4_b791_0858b325e0c0.slice/crio-4e66106fcff0e63fdfe70388a6ff22ccb3a7312d6f04cbd62e88c455be4ae03f WatchSource:0}: Error finding container 4e66106fcff0e63fdfe70388a6ff22ccb3a7312d6f04cbd62e88c455be4ae03f: Status 404 returned error can't find the container with id 4e66106fcff0e63fdfe70388a6ff22ccb3a7312d6f04cbd62e88c455be4ae03f
Sep 10 13:35:06.359273 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.359237    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"]
Sep 10 13:35:06.362889 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.362868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:06.367082 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.367053    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d6159c-dockercfg-vjh6b\""
Sep 10 13:35:06.368292 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.368270    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d6159c\""
Sep 10 13:35:06.382410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.382334    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"]
Sep 10 13:35:06.510024 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.509983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd8zn\" (UniqueName: \"kubernetes.io/projected/fdc038d6-c745-46a0-bed1-ddca6e90470a-kube-api-access-fd8zn\") pod \"payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w\" (UID: \"fdc038d6-c745-46a0-bed1-ddca6e90470a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:06.510230 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.510173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc038d6-c745-46a0-bed1-ddca6e90470a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w\" (UID: \"fdc038d6-c745-46a0-bed1-ddca6e90470a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:06.596671 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.596637    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk" event={"ID":"325c1350-25a8-4bc4-b791-0858b325e0c0","Type":"ContainerStarted","Data":"0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92"}
Sep 10 13:35:06.596879 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.596689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk" event={"ID":"325c1350-25a8-4bc4-b791-0858b325e0c0","Type":"ContainerStarted","Data":"4e66106fcff0e63fdfe70388a6ff22ccb3a7312d6f04cbd62e88c455be4ae03f"}
Sep 10 13:35:06.596879 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.596753    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:06.610668 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.610636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc038d6-c745-46a0-bed1-ddca6e90470a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w\" (UID: \"fdc038d6-c745-46a0-bed1-ddca6e90470a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:06.610840 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.610682    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd8zn\" (UniqueName: \"kubernetes.io/projected/fdc038d6-c745-46a0-bed1-ddca6e90470a-kube-api-access-fd8zn\") pod \"payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w\" (UID: \"fdc038d6-c745-46a0-bed1-ddca6e90470a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:06.611401 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.611380    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc038d6-c745-46a0-bed1-ddca6e90470a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w\" (UID: \"fdc038d6-c745-46a0-bed1-ddca6e90470a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:06.628309 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.628281    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd8zn\" (UniqueName: \"kubernetes.io/projected/fdc038d6-c745-46a0-bed1-ddca6e90470a-kube-api-access-fd8zn\") pod \"payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w\" (UID: \"fdc038d6-c745-46a0-bed1-ddca6e90470a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:06.645787 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.645687    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk" podStartSLOduration=1.25927106 podStartE2EDuration="1.645668981s" totalImagesPullingTime="386.397921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:06.643859626 +0000 UTC m=+1011.660691601" watchObservedRunningTime="2026-09-10 13:35:06.645668981 +0000 UTC m=+1011.662500951"
Sep 10 13:35:06.673549 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.673497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:06.962495 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:06.962466    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"]
Sep 10 13:35:06.963109 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:35:06.963078    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc038d6_c745_46a0_bed1_ddca6e90470a.slice/crio-71c679e24ec067c3f69b8c7d59c6254d0dd34ac20941d6d05548806afe88ec20 WatchSource:0}: Error finding container 71c679e24ec067c3f69b8c7d59c6254d0dd34ac20941d6d05548806afe88ec20: Status 404 returned error can't find the container with id 71c679e24ec067c3f69b8c7d59c6254d0dd34ac20941d6d05548806afe88ec20
Sep 10 13:35:07.605294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:07.605260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w" event={"ID":"fdc038d6-c745-46a0-bed1-ddca6e90470a","Type":"ContainerStarted","Data":"62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9"}
Sep 10 13:35:07.605781 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:07.605315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w" event={"ID":"fdc038d6-c745-46a0-bed1-ddca6e90470a","Type":"ContainerStarted","Data":"71c679e24ec067c3f69b8c7d59c6254d0dd34ac20941d6d05548806afe88ec20"}
Sep 10 13:35:07.605781 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:07.605357    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:07.634532 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:07.634470    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w" podStartSLOduration=1.634446262 podStartE2EDuration="1.634446262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:07.632499533 +0000 UTC m=+1012.649331507" watchObservedRunningTime="2026-09-10 13:35:07.634446262 +0000 UTC m=+1012.651278232"
Sep 10 13:35:12.612445 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:12.612414    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:35:18.612233 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:18.612183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:35:28.377992 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.377955    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"]
Sep 10 13:35:28.378496 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.378165    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg" podUID="36527a4b-aba2-4291-9f28-7ff0589aef0e" containerName="maas-api" containerID="cri-o://57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a" gracePeriod=30
Sep 10 13:35:28.643453 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.643393    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:35:28.710277 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.710248    2578 generic.go:356] "Generic (PLEG): container finished" podID="36527a4b-aba2-4291-9f28-7ff0589aef0e" containerID="57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a" exitCode=0
Sep 10 13:35:28.710428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.710297    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"
Sep 10 13:35:28.710428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.710333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg" event={"ID":"36527a4b-aba2-4291-9f28-7ff0589aef0e","Type":"ContainerDied","Data":"57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a"}
Sep 10 13:35:28.710428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.710389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg" event={"ID":"36527a4b-aba2-4291-9f28-7ff0589aef0e","Type":"ContainerDied","Data":"5219a39157481c95f9f5cb6e52608ad2edce3221ec42a8609f40a9134f125261"}
Sep 10 13:35:28.710428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.710418    2578 scope.go:117] "RemoveContainer" containerID="57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a"
Sep 10 13:35:28.719109 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.719083    2578 scope.go:117] "RemoveContainer" containerID="57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a"
Sep 10 13:35:28.719371 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.719353    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36527a4b-aba2-4291-9f28-7ff0589aef0e-openshift-service-ca\") pod \"36527a4b-aba2-4291-9f28-7ff0589aef0e\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") "
Sep 10 13:35:28.719477 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:35:28.719383    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a\": container with ID starting with 57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a not found: ID does not exist" containerID="57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a"
Sep 10 13:35:28.719477 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.719418    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a"} err="failed to get container status \"57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a\": rpc error: code = NotFound desc = could not find container \"57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a\": container with ID starting with 57497273658b2bfadcbe1282175ec935649ca770974b62210f9f9e614d63908a not found: ID does not exist"
Sep 10 13:35:28.719477 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.719391    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-maas-api-tls\") pod \"36527a4b-aba2-4291-9f28-7ff0589aef0e\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") "
Sep 10 13:35:28.719656 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.719491    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-metrics-certs\") pod \"36527a4b-aba2-4291-9f28-7ff0589aef0e\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") "
Sep 10 13:35:28.719656 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.719553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkbgq\" (UniqueName: \"kubernetes.io/projected/36527a4b-aba2-4291-9f28-7ff0589aef0e-kube-api-access-jkbgq\") pod \"36527a4b-aba2-4291-9f28-7ff0589aef0e\" (UID: \"36527a4b-aba2-4291-9f28-7ff0589aef0e\") "
Sep 10 13:35:28.720076 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.719817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36527a4b-aba2-4291-9f28-7ff0589aef0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36527a4b-aba2-4291-9f28-7ff0589aef0e" (UID: "36527a4b-aba2-4291-9f28-7ff0589aef0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:35:28.720076 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.720035    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36527a4b-aba2-4291-9f28-7ff0589aef0e-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:35:28.721720 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.721693    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36527a4b-aba2-4291-9f28-7ff0589aef0e" (UID: "36527a4b-aba2-4291-9f28-7ff0589aef0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:35:28.721870 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.721848    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36527a4b-aba2-4291-9f28-7ff0589aef0e" (UID: "36527a4b-aba2-4291-9f28-7ff0589aef0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:35:28.721921 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.721866    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36527a4b-aba2-4291-9f28-7ff0589aef0e-kube-api-access-jkbgq" (OuterVolumeSpecName: "kube-api-access-jkbgq") pod "36527a4b-aba2-4291-9f28-7ff0589aef0e" (UID: "36527a4b-aba2-4291-9f28-7ff0589aef0e"). InnerVolumeSpecName "kube-api-access-jkbgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:35:28.821432 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.821398    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkbgq\" (UniqueName: \"kubernetes.io/projected/36527a4b-aba2-4291-9f28-7ff0589aef0e-kube-api-access-jkbgq\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:35:28.821432 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.821425    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:35:28.821432 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:28.821434    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36527a4b-aba2-4291-9f28-7ff0589aef0e-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:35:29.045870 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:29.045838    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"]
Sep 10 13:35:29.047653 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:29.047632    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-3f02bf87-6b4f8447df-66fvg"]
Sep 10 13:35:29.452331 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:29.452254    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36527a4b-aba2-4291-9f28-7ff0589aef0e" path="/var/lib/kubelet/pods/36527a4b-aba2-4291-9f28-7ff0589aef0e/volumes"
Sep 10 13:35:32.518210 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.518167    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"]
Sep 10 13:35:32.518600 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.518586    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36527a4b-aba2-4291-9f28-7ff0589aef0e" containerName="maas-api"
Sep 10 13:35:32.518600 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.518600    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="36527a4b-aba2-4291-9f28-7ff0589aef0e" containerName="maas-api"
Sep 10 13:35:32.518678 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.518669    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="36527a4b-aba2-4291-9f28-7ff0589aef0e" containerName="maas-api"
Sep 10 13:35:32.523189 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.523171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.528315 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.528288    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-deba50\"/\"autores-model-deba50-kserve-self-signed-certs\""
Sep 10 13:35:32.528929 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.528902    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-deba50\"/\"openshift-service-ca.crt\""
Sep 10 13:35:32.528929 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.528918    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-deba50\"/\"default-dockercfg-rzcfj\""
Sep 10 13:35:32.528929 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.528902    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-deba50\"/\"kube-root-ca.crt\""
Sep 10 13:35:32.556380 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.556348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48af339-9791-4f9b-87c9-964e2e0b84d9-tls-certs\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.556567 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.556420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx4zz\" (UniqueName: \"kubernetes.io/projected/e48af339-9791-4f9b-87c9-964e2e0b84d9-kube-api-access-wx4zz\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.556567 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.556510    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-home\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.556567 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.556541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-dshm\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.556698 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.556581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-tmp-dir\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.556698 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.556605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-model-cache\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.560014 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.559986    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"]
Sep 10 13:35:32.657997 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.657960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48af339-9791-4f9b-87c9-964e2e0b84d9-tls-certs\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.657997 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.658009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx4zz\" (UniqueName: \"kubernetes.io/projected/e48af339-9791-4f9b-87c9-964e2e0b84d9-kube-api-access-wx4zz\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.658294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.658060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-home\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.658294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.658076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-dshm\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.658294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.658109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-tmp-dir\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.658294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.658127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-model-cache\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.658560 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.658540    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-home\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.658622 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.658582    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-model-cache\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.658672 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.658648    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-tmp-dir\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.660403 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.660381    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-dshm\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.660692 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.660673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48af339-9791-4f9b-87c9-964e2e0b84d9-tls-certs\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.669802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.669768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx4zz\" (UniqueName: \"kubernetes.io/projected/e48af339-9791-4f9b-87c9-964e2e0b84d9-kube-api-access-wx4zz\") pod \"autores-model-deba50-kserve-59bb9b9dbc-wc9hx\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") " pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:32.832832 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:32.832738    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:33.117639 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.117603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"]
Sep 10 13:35:33.119092 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:35:33.119068    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode48af339_9791_4f9b_87c9_964e2e0b84d9.slice/crio-bc273573d2080938ec700e52f1be9b23c43acd4330a3f92b2917c108d8f7eb97 WatchSource:0}: Error finding container bc273573d2080938ec700e52f1be9b23c43acd4330a3f92b2917c108d8f7eb97: Status 404 returned error can't find the container with id bc273573d2080938ec700e52f1be9b23c43acd4330a3f92b2917c108d8f7eb97
Sep 10 13:35:33.392509 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.392425    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"]
Sep 10 13:35:33.396463 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.396437    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.399919 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.399892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4907abfd-serving-cert\""
Sep 10 13:35:33.413527 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.413340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"]
Sep 10 13:35:33.465588 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.465546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-openshift-service-ca\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.465806 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.465635    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcj5h\" (UniqueName: \"kubernetes.io/projected/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-kube-api-access-gcj5h\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.465806 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.465668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-metrics-certs\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.466049 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.466011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-maas-api-tls\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.567351 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.567307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-maas-api-tls\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.567818 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.567417    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-openshift-service-ca\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.567818 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:35:33.567469    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-serving-cert: secret "maas-api-e2e-migrate-4907abfd-serving-cert" not found
Sep 10 13:35:33.567818 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.567482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcj5h\" (UniqueName: \"kubernetes.io/projected/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-kube-api-access-gcj5h\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.567818 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.567512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-metrics-certs\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.567818 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:35:33.567548    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-maas-api-tls podName:4ea1b3d5-7461-4e1f-99fe-f4616d9ee233 nodeName:}" failed. No retries permitted until 2026-09-10 13:35:34.067524683 +0000 UTC m=+1039.084356647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-maas-api-tls") pod "maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8" (UID: "4ea1b3d5-7461-4e1f-99fe-f4616d9ee233") : secret "maas-api-e2e-migrate-4907abfd-serving-cert" not found
Sep 10 13:35:33.568163 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.568141    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-openshift-service-ca\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.569915 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.569893    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-metrics-certs\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.588265 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.588231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcj5h\" (UniqueName: \"kubernetes.io/projected/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-kube-api-access-gcj5h\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:33.740542 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.740488    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx" event={"ID":"e48af339-9791-4f9b-87c9-964e2e0b84d9","Type":"ContainerStarted","Data":"991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a"}
Sep 10 13:35:33.740746 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.740567    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx" event={"ID":"e48af339-9791-4f9b-87c9-964e2e0b84d9","Type":"ContainerStarted","Data":"bc273573d2080938ec700e52f1be9b23c43acd4330a3f92b2917c108d8f7eb97"}
Sep 10 13:35:33.771016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:33.770957    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx" podStartSLOduration=1.770941487 podStartE2EDuration="1.770941487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:33.767407051 +0000 UTC m=+1038.784239022" watchObservedRunningTime="2026-09-10 13:35:33.770941487 +0000 UTC m=+1038.787773458"
Sep 10 13:35:34.072601 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:34.072528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-maas-api-tls\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:34.074888 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:34.074865    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-maas-api-tls\") pod \"maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:34.308092 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:34.308055    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:34.593704 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:34.593669    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"]
Sep 10 13:35:34.596308 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:35:34.596278    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea1b3d5_7461_4e1f_99fe_f4616d9ee233.slice/crio-25d9ede92a1aba99318e2ee09709c8c12cd92b39c400c1d8ec59e1dae449e3e8 WatchSource:0}: Error finding container 25d9ede92a1aba99318e2ee09709c8c12cd92b39c400c1d8ec59e1dae449e3e8: Status 404 returned error can't find the container with id 25d9ede92a1aba99318e2ee09709c8c12cd92b39c400c1d8ec59e1dae449e3e8
Sep 10 13:35:34.746062 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:34.746025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8" event={"ID":"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233","Type":"ContainerStarted","Data":"25d9ede92a1aba99318e2ee09709c8c12cd92b39c400c1d8ec59e1dae449e3e8"}
Sep 10 13:35:35.752552 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:35.752512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8" event={"ID":"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233","Type":"ContainerStarted","Data":"8e6f6abf921ca2b94fd965f7c5cdd7beb511630e646079784b52e27aa466f47f"}
Sep 10 13:35:35.752908 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:35.752645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:35.782760 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:35.782709    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8" podStartSLOduration=2.354363757 podStartE2EDuration="2.782693747s" totalImagesPullingTime="428.32999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:35:35.780720687 +0000 UTC m=+1040.797552657" watchObservedRunningTime="2026-09-10 13:35:35.782693747 +0000 UTC m=+1040.799525718"
Sep 10 13:35:41.763311 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:41.763276    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:35:42.832963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:42.832926    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:42.832963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:42.832974    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:42.845569 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:42.845546    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:35:43.804640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:35:43.804610    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:36:03.956591 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:03.956551    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"]
Sep 10 13:36:14.694674 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:14.694595    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"]
Sep 10 13:36:14.695067 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:14.694812    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8" podUID="4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" containerName="maas-api" containerID="cri-o://8e6f6abf921ca2b94fd965f7c5cdd7beb511630e646079784b52e27aa466f47f" gracePeriod=30
Sep 10 13:36:14.937335 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:14.937297    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"]
Sep 10 13:36:14.937530 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:14.937496    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj" podUID="2e209893-06bb-43b0-a87e-29ffaef79ef2" containerName="maas-api" containerID="cri-o://ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a" gracePeriod=30
Sep 10 13:36:14.954858 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:14.954828    2578 generic.go:356] "Generic (PLEG): container finished" podID="4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" containerID="8e6f6abf921ca2b94fd965f7c5cdd7beb511630e646079784b52e27aa466f47f" exitCode=0
Sep 10 13:36:14.954999 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:14.954900    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8" event={"ID":"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233","Type":"ContainerDied","Data":"8e6f6abf921ca2b94fd965f7c5cdd7beb511630e646079784b52e27aa466f47f"}
Sep 10 13:36:14.954999 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:14.954973    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8" event={"ID":"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233","Type":"ContainerDied","Data":"25d9ede92a1aba99318e2ee09709c8c12cd92b39c400c1d8ec59e1dae449e3e8"}
Sep 10 13:36:14.955081 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:14.955002    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25d9ede92a1aba99318e2ee09709c8c12cd92b39c400c1d8ec59e1dae449e3e8"
Sep 10 13:36:15.027303 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.027267    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:36:15.035062 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.035033    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"]
Sep 10 13:36:15.035263 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.035241    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm" podUID="8d2fc527-6f9f-4b48-94db-b2ae5008e5a2" containerName="payload-pre-processing" containerID="cri-o://6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5" gracePeriod=30
Sep 10 13:36:15.051712 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.051680    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcj5h\" (UniqueName: \"kubernetes.io/projected/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-kube-api-access-gcj5h\") pod \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") "
Sep 10 13:36:15.051859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.051759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-metrics-certs\") pod \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") "
Sep 10 13:36:15.051859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.051802    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-openshift-service-ca\") pod \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") "
Sep 10 13:36:15.051859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.051826    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-maas-api-tls\") pod \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\" (UID: \"4ea1b3d5-7461-4e1f-99fe-f4616d9ee233\") "
Sep 10 13:36:15.053359 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.053171    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" (UID: "4ea1b3d5-7461-4e1f-99fe-f4616d9ee233"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:36:15.056675 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.056570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-kube-api-access-gcj5h" (OuterVolumeSpecName: "kube-api-access-gcj5h") pod "4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" (UID: "4ea1b3d5-7461-4e1f-99fe-f4616d9ee233"). InnerVolumeSpecName "kube-api-access-gcj5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:36:15.056675 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.056634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" (UID: "4ea1b3d5-7461-4e1f-99fe-f4616d9ee233"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:36:15.058020 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.057983    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" (UID: "4ea1b3d5-7461-4e1f-99fe-f4616d9ee233"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:36:15.152778 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.152749    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcj5h\" (UniqueName: \"kubernetes.io/projected/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-kube-api-access-gcj5h\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.152778 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.152779    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.152778 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.152793    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.153039 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.152805    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.217665 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.217630    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:36:15.254095 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.254064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-maas-api-tls\") pod \"2e209893-06bb-43b0-a87e-29ffaef79ef2\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") "
Sep 10 13:36:15.254262 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.254145    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-metrics-certs\") pod \"2e209893-06bb-43b0-a87e-29ffaef79ef2\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") "
Sep 10 13:36:15.254262 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.254185    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e209893-06bb-43b0-a87e-29ffaef79ef2-openshift-service-ca\") pod \"2e209893-06bb-43b0-a87e-29ffaef79ef2\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") "
Sep 10 13:36:15.254262 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.254246    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62lhc\" (UniqueName: \"kubernetes.io/projected/2e209893-06bb-43b0-a87e-29ffaef79ef2-kube-api-access-62lhc\") pod \"2e209893-06bb-43b0-a87e-29ffaef79ef2\" (UID: \"2e209893-06bb-43b0-a87e-29ffaef79ef2\") "
Sep 10 13:36:15.254616 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.254576    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e209893-06bb-43b0-a87e-29ffaef79ef2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2e209893-06bb-43b0-a87e-29ffaef79ef2" (UID: "2e209893-06bb-43b0-a87e-29ffaef79ef2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:36:15.256903 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.256867    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e209893-06bb-43b0-a87e-29ffaef79ef2-kube-api-access-62lhc" (OuterVolumeSpecName: "kube-api-access-62lhc") pod "2e209893-06bb-43b0-a87e-29ffaef79ef2" (UID: "2e209893-06bb-43b0-a87e-29ffaef79ef2"). InnerVolumeSpecName "kube-api-access-62lhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:36:15.257686 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.256910    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2e209893-06bb-43b0-a87e-29ffaef79ef2" (UID: "2e209893-06bb-43b0-a87e-29ffaef79ef2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:36:15.257686 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.257021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2e209893-06bb-43b0-a87e-29ffaef79ef2" (UID: "2e209893-06bb-43b0-a87e-29ffaef79ef2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:36:15.306385 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.306358    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:36:15.355102 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.355072    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48rhd\" (UniqueName: \"kubernetes.io/projected/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-kube-api-access-48rhd\") pod \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\" (UID: \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\") "
Sep 10 13:36:15.355288 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.355233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-plugins-config-volume\") pod \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\" (UID: \"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2\") "
Sep 10 13:36:15.355425 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.355408    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.355425 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.355424    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e209893-06bb-43b0-a87e-29ffaef79ef2-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.355577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.355437    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e209893-06bb-43b0-a87e-29ffaef79ef2-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.355577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.355450    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62lhc\" (UniqueName: \"kubernetes.io/projected/2e209893-06bb-43b0-a87e-29ffaef79ef2-kube-api-access-62lhc\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.355707 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.355676    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d2fc527-6f9f-4b48-94db-b2ae5008e5a2" (UID: "8d2fc527-6f9f-4b48-94db-b2ae5008e5a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:36:15.357179 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.357156    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-kube-api-access-48rhd" (OuterVolumeSpecName: "kube-api-access-48rhd") pod "8d2fc527-6f9f-4b48-94db-b2ae5008e5a2" (UID: "8d2fc527-6f9f-4b48-94db-b2ae5008e5a2"). InnerVolumeSpecName "kube-api-access-48rhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:36:15.456071 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.456039    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.456071 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.456072    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rhd\" (UniqueName: \"kubernetes.io/projected/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2-kube-api-access-48rhd\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:15.962547 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.962512    2578 generic.go:356] "Generic (PLEG): container finished" podID="2e209893-06bb-43b0-a87e-29ffaef79ef2" containerID="ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a" exitCode=0
Sep 10 13:36:15.963037 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.962584    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"
Sep 10 13:36:15.963037 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.962584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj" event={"ID":"2e209893-06bb-43b0-a87e-29ffaef79ef2","Type":"ContainerDied","Data":"ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a"}
Sep 10 13:36:15.963037 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.962720    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj" event={"ID":"2e209893-06bb-43b0-a87e-29ffaef79ef2","Type":"ContainerDied","Data":"585f9f78274333c83f6878e2017f39620232530d552647b7b0ab6de7aae18812"}
Sep 10 13:36:15.963037 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.962753    2578 scope.go:117] "RemoveContainer" containerID="ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a"
Sep 10 13:36:15.964054 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.964035    2578 generic.go:356] "Generic (PLEG): container finished" podID="8d2fc527-6f9f-4b48-94db-b2ae5008e5a2" containerID="6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5" exitCode=0
Sep 10 13:36:15.964136 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.964061    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm" event={"ID":"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2","Type":"ContainerDied","Data":"6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5"}
Sep 10 13:36:15.964136 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.964099    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"
Sep 10 13:36:15.964136 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.964110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm" event={"ID":"8d2fc527-6f9f-4b48-94db-b2ae5008e5a2","Type":"ContainerDied","Data":"3220a35f8640fbc302cd8aaf02089d461269e21e5dc1df576460c0dc4006e7ce"}
Sep 10 13:36:15.964363 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.964345    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"
Sep 10 13:36:15.972083 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.972062    2578 scope.go:117] "RemoveContainer" containerID="ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a"
Sep 10 13:36:15.972404 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:36:15.972379    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a\": container with ID starting with ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a not found: ID does not exist" containerID="ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a"
Sep 10 13:36:15.972515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.972412    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a"} err="failed to get container status \"ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a\": rpc error: code = NotFound desc = could not find container \"ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a\": container with ID starting with ff5f00410b7ab13d6fc936bbf9d8c9c55f58c1bd9e64d57e30c2502cd6a2517a not found: ID does not exist"
Sep 10 13:36:15.972515 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.972435    2578 scope.go:117] "RemoveContainer" containerID="6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5"
Sep 10 13:36:15.981418 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.981394    2578 scope.go:117] "RemoveContainer" containerID="6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5"
Sep 10 13:36:15.981800 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:36:15.981773    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5\": container with ID starting with 6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5 not found: ID does not exist" containerID="6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5"
Sep 10 13:36:15.981917 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:15.981809    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5"} err="failed to get container status \"6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5\": rpc error: code = NotFound desc = could not find container \"6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5\": container with ID starting with 6f497f9fcfb239b17023a7d77f62637871e7de0204cb92249f5145222e0a85b5 not found: ID does not exist"
Sep 10 13:36:16.008539 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:16.008502    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"]
Sep 10 13:36:16.010805 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:16.010775    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-deba50-7d7fcc98ff-ls8zj"]
Sep 10 13:36:16.032820 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:16.032790    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"]
Sep 10 13:36:16.037244 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:16.037215    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-deba50-59864b8b6b-hwbjm"]
Sep 10 13:36:16.057005 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:16.056972    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"]
Sep 10 13:36:16.063712 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:16.063686    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4907abfd-67bcfdbc57-n8fx8"]
Sep 10 13:36:17.452061 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:17.452026    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e209893-06bb-43b0-a87e-29ffaef79ef2" path="/var/lib/kubelet/pods/2e209893-06bb-43b0-a87e-29ffaef79ef2/volumes"
Sep 10 13:36:17.452464 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:17.452448    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" path="/var/lib/kubelet/pods/4ea1b3d5-7461-4e1f-99fe-f4616d9ee233/volumes"
Sep 10 13:36:17.452757 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:17.452745    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d2fc527-6f9f-4b48-94db-b2ae5008e5a2" path="/var/lib/kubelet/pods/8d2fc527-6f9f-4b48-94db-b2ae5008e5a2/volumes"
Sep 10 13:36:18.976061 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:18.976020    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx" podUID="e48af339-9791-4f9b-87c9-964e2e0b84d9" containerName="main" containerID="cri-o://991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a" gracePeriod=45
Sep 10 13:36:19.234747 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.234722    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:36:19.290184 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290152    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-tmp-dir\") pod \"e48af339-9791-4f9b-87c9-964e2e0b84d9\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") "
Sep 10 13:36:19.290409 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-model-cache\") pod \"e48af339-9791-4f9b-87c9-964e2e0b84d9\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") "
Sep 10 13:36:19.290409 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48af339-9791-4f9b-87c9-964e2e0b84d9-tls-certs\") pod \"e48af339-9791-4f9b-87c9-964e2e0b84d9\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") "
Sep 10 13:36:19.290409 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx4zz\" (UniqueName: \"kubernetes.io/projected/e48af339-9791-4f9b-87c9-964e2e0b84d9-kube-api-access-wx4zz\") pod \"e48af339-9791-4f9b-87c9-964e2e0b84d9\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") "
Sep 10 13:36:19.290409 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-dshm\") pod \"e48af339-9791-4f9b-87c9-964e2e0b84d9\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") "
Sep 10 13:36:19.290409 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290380    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-home\") pod \"e48af339-9791-4f9b-87c9-964e2e0b84d9\" (UID: \"e48af339-9791-4f9b-87c9-964e2e0b84d9\") "
Sep 10 13:36:19.290665 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290514    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e48af339-9791-4f9b-87c9-964e2e0b84d9" (UID: "e48af339-9791-4f9b-87c9-964e2e0b84d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:36:19.290665 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290557    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "e48af339-9791-4f9b-87c9-964e2e0b84d9" (UID: "e48af339-9791-4f9b-87c9-964e2e0b84d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:36:19.290763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290654    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-model-cache\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:19.290763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290697    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-tmp-dir\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:19.290763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.290738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-home" (OuterVolumeSpecName: "home") pod "e48af339-9791-4f9b-87c9-964e2e0b84d9" (UID: "e48af339-9791-4f9b-87c9-964e2e0b84d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:36:19.293144 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.293118    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-dshm" (OuterVolumeSpecName: "dshm") pod "e48af339-9791-4f9b-87c9-964e2e0b84d9" (UID: "e48af339-9791-4f9b-87c9-964e2e0b84d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:36:19.293321 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.293208    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e48af339-9791-4f9b-87c9-964e2e0b84d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e48af339-9791-4f9b-87c9-964e2e0b84d9" (UID: "e48af339-9791-4f9b-87c9-964e2e0b84d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:36:19.294398 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.294363    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48af339-9791-4f9b-87c9-964e2e0b84d9-kube-api-access-wx4zz" (OuterVolumeSpecName: "kube-api-access-wx4zz") pod "e48af339-9791-4f9b-87c9-964e2e0b84d9" (UID: "e48af339-9791-4f9b-87c9-964e2e0b84d9"). InnerVolumeSpecName "kube-api-access-wx4zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:36:19.391445 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.391406    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e48af339-9791-4f9b-87c9-964e2e0b84d9-tls-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:19.391445 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.391439    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx4zz\" (UniqueName: \"kubernetes.io/projected/e48af339-9791-4f9b-87c9-964e2e0b84d9-kube-api-access-wx4zz\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:19.391445 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.391449    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-dshm\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:19.391445 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.391457    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e48af339-9791-4f9b-87c9-964e2e0b84d9-home\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:19.986974 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.986936    2578 generic.go:356] "Generic (PLEG): container finished" podID="e48af339-9791-4f9b-87c9-964e2e0b84d9" containerID="991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a" exitCode=0
Sep 10 13:36:19.987491 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.986984    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"
Sep 10 13:36:19.987491 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.987002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx" event={"ID":"e48af339-9791-4f9b-87c9-964e2e0b84d9","Type":"ContainerDied","Data":"991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a"}
Sep 10 13:36:19.987491 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.987040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx" event={"ID":"e48af339-9791-4f9b-87c9-964e2e0b84d9","Type":"ContainerDied","Data":"bc273573d2080938ec700e52f1be9b23c43acd4330a3f92b2917c108d8f7eb97"}
Sep 10 13:36:19.987491 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.987059    2578 scope.go:117] "RemoveContainer" containerID="991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a"
Sep 10 13:36:19.995371 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.995353    2578 scope.go:117] "RemoveContainer" containerID="991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a"
Sep 10 13:36:19.995683 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:36:19.995663    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a\": container with ID starting with 991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a not found: ID does not exist" containerID="991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a"
Sep 10 13:36:19.995734 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:19.995693    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a"} err="failed to get container status \"991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a\": rpc error: code = NotFound desc = could not find container \"991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a\": container with ID starting with 991440562db77da270a9573e59236687dd1f2446e4c03384c330d1c83408053a not found: ID does not exist"
Sep 10 13:36:20.012506 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.012459    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"]
Sep 10 13:36:20.019599 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.019566    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-deba50/autores-model-deba50-kserve-59bb9b9dbc-wc9hx"]
Sep 10 13:36:20.491567 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.491535    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"]
Sep 10 13:36:20.491949 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.491933    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2fc527-6f9f-4b48-94db-b2ae5008e5a2" containerName="payload-pre-processing"
Sep 10 13:36:20.491949 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.491949    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2fc527-6f9f-4b48-94db-b2ae5008e5a2" containerName="payload-pre-processing"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.491959    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e209893-06bb-43b0-a87e-29ffaef79ef2" containerName="maas-api"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.491965    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e209893-06bb-43b0-a87e-29ffaef79ef2" containerName="maas-api"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.491983    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48af339-9791-4f9b-87c9-964e2e0b84d9" containerName="main"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.491989    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48af339-9791-4f9b-87c9-964e2e0b84d9" containerName="main"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.491995    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" containerName="maas-api"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.492001    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" containerName="maas-api"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.492057    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d2fc527-6f9f-4b48-94db-b2ae5008e5a2" containerName="payload-pre-processing"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.492066    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48af339-9791-4f9b-87c9-964e2e0b84d9" containerName="main"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.492073    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea1b3d5-7461-4e1f-99fe-f4616d9ee233" containerName="maas-api"
Sep 10 13:36:20.492113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.492078    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e209893-06bb-43b0-a87e-29ffaef79ef2" containerName="maas-api"
Sep 10 13:36:20.496872 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.496845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.499412 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.499384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-95ae6288-serving-cert\""
Sep 10 13:36:20.511695 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.511662    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"]
Sep 10 13:36:20.601759 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.601723    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-openshift-service-ca\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.601759 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.601763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-metrics-certs\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.602024 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.601812    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4rp\" (UniqueName: \"kubernetes.io/projected/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-kube-api-access-bx4rp\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.602024 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.601900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-maas-api-tls\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.702960 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.702912    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-maas-api-tls\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.703163 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.702988    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-openshift-service-ca\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.703163 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.703019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-metrics-certs\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.703163 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.703082    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4rp\" (UniqueName: \"kubernetes.io/projected/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-kube-api-access-bx4rp\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.703163 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:36:20.703084    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-serving-cert: secret "maas-api-e2e-ait-95ae6288-serving-cert" not found
Sep 10 13:36:20.703432 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:36:20.703174    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-maas-api-tls podName:5cd3d037-f52b-4a5c-be6a-9eb1ad995425 nodeName:}" failed. No retries permitted until 2026-09-10 13:36:21.203150021 +0000 UTC m=+1086.219981986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-maas-api-tls") pod "maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks" (UID: "5cd3d037-f52b-4a5c-be6a-9eb1ad995425") : secret "maas-api-e2e-ait-95ae6288-serving-cert" not found
Sep 10 13:36:20.704061 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.704040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-openshift-service-ca\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.706201 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.706176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-metrics-certs\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:20.721758 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:20.721729    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4rp\" (UniqueName: \"kubernetes.io/projected/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-kube-api-access-bx4rp\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:21.208426 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:21.208387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-maas-api-tls\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:21.210885 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:21.210847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-maas-api-tls\") pod \"maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:21.410664 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:21.410623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:21.454380 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:21.454345    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e48af339-9791-4f9b-87c9-964e2e0b84d9" path="/var/lib/kubelet/pods/e48af339-9791-4f9b-87c9-964e2e0b84d9/volumes"
Sep 10 13:36:21.697957 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:21.697933    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"]
Sep 10 13:36:21.699837 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:36:21.699809    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd3d037_f52b_4a5c_be6a_9eb1ad995425.slice/crio-dca2dd049fd4c7bc938f24392306655514dac4e7a5d38d698f31f83d8abaa44d WatchSource:0}: Error finding container dca2dd049fd4c7bc938f24392306655514dac4e7a5d38d698f31f83d8abaa44d: Status 404 returned error can't find the container with id dca2dd049fd4c7bc938f24392306655514dac4e7a5d38d698f31f83d8abaa44d
Sep 10 13:36:22.002310 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:22.002263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks" event={"ID":"5cd3d037-f52b-4a5c-be6a-9eb1ad995425","Type":"ContainerStarted","Data":"dca2dd049fd4c7bc938f24392306655514dac4e7a5d38d698f31f83d8abaa44d"}
Sep 10 13:36:23.009122 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:23.009086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks" event={"ID":"5cd3d037-f52b-4a5c-be6a-9eb1ad995425","Type":"ContainerStarted","Data":"1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254"}
Sep 10 13:36:23.009552 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:23.009230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:23.044049 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:23.043989    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks" podStartSLOduration=2.582624342 podStartE2EDuration="3.043972465s" totalImagesPullingTime="461.348123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:23.042546429 +0000 UTC m=+1088.059378401" watchObservedRunningTime="2026-09-10 13:36:23.043972465 +0000 UTC m=+1088.060804437"
Sep 10 13:36:24.412727 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.412689    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"]
Sep 10 13:36:24.416362 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.416341    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.423495 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.423456    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bc03be\"/\"test-model-bc03be-kserve-self-signed-certs\""
Sep 10 13:36:24.423640 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.423622    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bc03be\"/\"kube-root-ca.crt\""
Sep 10 13:36:24.423749 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.423643    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bc03be\"/\"openshift-service-ca.crt\""
Sep 10 13:36:24.423836 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.423764    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bc03be\"/\"default-dockercfg-szkmn\""
Sep 10 13:36:24.446077 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.446038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"]
Sep 10 13:36:24.543055 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.543020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tls-certs\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.543252 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.543086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-model-cache\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.543252 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.543120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-home\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.543252 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.543179    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxjt\" (UniqueName: \"kubernetes.io/projected/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-kube-api-access-wpxjt\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.543436 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.543296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tmp-dir\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.543436 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.543351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-dshm\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.644459 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.644420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tmp-dir\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.644638 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.644471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-dshm\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.644638 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.644507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tls-certs\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.644638 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.644545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-model-cache\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.644638 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.644577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-home\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.644638 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.644599    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxjt\" (UniqueName: \"kubernetes.io/projected/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-kube-api-access-wpxjt\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.644947 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.644916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tmp-dir\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.645098 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.644957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-home\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.645098 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.645023    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-model-cache\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.647428 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.647405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-dshm\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.647568 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.647553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tls-certs\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.659596 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.659563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxjt\" (UniqueName: \"kubernetes.io/projected/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-kube-api-access-wpxjt\") pod \"test-model-bc03be-kserve-5cbcb49c4c-sf6zp\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") " pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:24.725690 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:24.725586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:25.047757 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:36:25.047693    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda71c4bf3_f3a5_47f2_9a4f_59ee869c282d.slice/crio-b0e34cb66ceed703725bfe4e4eda7c4a876d629172a5308adb4aaae3891ba8cd WatchSource:0}: Error finding container b0e34cb66ceed703725bfe4e4eda7c4a876d629172a5308adb4aaae3891ba8cd: Status 404 returned error can't find the container with id b0e34cb66ceed703725bfe4e4eda7c4a876d629172a5308adb4aaae3891ba8cd
Sep 10 13:36:25.052744 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:25.052714    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"]
Sep 10 13:36:26.025900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:26.025866    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp" event={"ID":"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d","Type":"ContainerStarted","Data":"d2ff2de88acb3fc9b9f4784e4d379092f45356e8fc7b52c43c0f2a3125bb5ad5"}
Sep 10 13:36:26.025900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:26.025906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp" event={"ID":"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d","Type":"ContainerStarted","Data":"b0e34cb66ceed703725bfe4e4eda7c4a876d629172a5308adb4aaae3891ba8cd"}
Sep 10 13:36:26.050658 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:26.050601    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp" podStartSLOduration=2.050582226 podStartE2EDuration="2.050582226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:26.04848924 +0000 UTC m=+1091.065321213" watchObservedRunningTime="2026-09-10 13:36:26.050582226 +0000 UTC m=+1091.067414198"
Sep 10 13:36:29.024601 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:29.024559    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:30.590981 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.590947    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"]
Sep 10 13:36:30.595244 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.595216    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.598024 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.597999    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-3c32b1-serving-cert\""
Sep 10 13:36:30.610183 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.610151    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"]
Sep 10 13:36:30.700739 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.700695    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-openshift-service-ca\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.700946 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.700844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wd2\" (UniqueName: \"kubernetes.io/projected/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-kube-api-access-94wd2\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.700946 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.700895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-metrics-certs\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.701062 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.700947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-maas-api-tls\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.802455 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.802414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-maas-api-tls\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.802648 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.802503    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-openshift-service-ca\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.802648 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.802577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wd2\" (UniqueName: \"kubernetes.io/projected/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-kube-api-access-94wd2\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.802648 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:36:30.802580    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-serving-cert: secret "maas-api-e2e-explicit-3c32b1-serving-cert" not found
Sep 10 13:36:30.802648 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.802610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-metrics-certs\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.802874 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:36:30.802678    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-maas-api-tls podName:17d7a5b0-ab91-4d67-a651-d1d7f24b33a1 nodeName:}" failed. No retries permitted until 2026-09-10 13:36:31.30265439 +0000 UTC m=+1096.319486360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-maas-api-tls") pod "maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd" (UID: "17d7a5b0-ab91-4d67-a651-d1d7f24b33a1") : secret "maas-api-e2e-explicit-3c32b1-serving-cert" not found
Sep 10 13:36:30.803305 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.803279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-openshift-service-ca\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.805117 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.805088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-metrics-certs\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:30.818183 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:30.818151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wd2\" (UniqueName: \"kubernetes.io/projected/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-kube-api-access-94wd2\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:31.307189 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.307147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-maas-api-tls\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:31.309819 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.309789    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-maas-api-tls\") pod \"maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:31.494085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.494048    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"]
Sep 10 13:36:31.498001 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.497975    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:31.502904 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.502866    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-3c32b1\""
Sep 10 13:36:31.508461 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.508433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:31.513094 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.513075    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-3c32b1-dockercfg-p59zr\""
Sep 10 13:36:31.525987 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.525957    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"]
Sep 10 13:36:31.610152 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.609990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x6v8\" (UniqueName: \"kubernetes.io/projected/b7173db0-0a67-497d-8e06-55bdb8a432e1-kube-api-access-2x6v8\") pod \"payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq\" (UID: \"b7173db0-0a67-497d-8e06-55bdb8a432e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:31.610152 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.610033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7173db0-0a67-497d-8e06-55bdb8a432e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq\" (UID: \"b7173db0-0a67-497d-8e06-55bdb8a432e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:31.711187 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.711148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x6v8\" (UniqueName: \"kubernetes.io/projected/b7173db0-0a67-497d-8e06-55bdb8a432e1-kube-api-access-2x6v8\") pod \"payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq\" (UID: \"b7173db0-0a67-497d-8e06-55bdb8a432e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:31.711187 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.711204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7173db0-0a67-497d-8e06-55bdb8a432e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq\" (UID: \"b7173db0-0a67-497d-8e06-55bdb8a432e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:31.711958 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.711937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7173db0-0a67-497d-8e06-55bdb8a432e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq\" (UID: \"b7173db0-0a67-497d-8e06-55bdb8a432e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:31.724872 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.724843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x6v8\" (UniqueName: \"kubernetes.io/projected/b7173db0-0a67-497d-8e06-55bdb8a432e1-kube-api-access-2x6v8\") pod \"payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq\" (UID: \"b7173db0-0a67-497d-8e06-55bdb8a432e1\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:31.805774 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.805738    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"]
Sep 10 13:36:31.812352 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:31.811605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:32.059707 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:32.059671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd" event={"ID":"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1","Type":"ContainerStarted","Data":"ff8743a27fedc4c6cd7ae3f3f91f18763cc88aef69a7b3e283ed37019b213c2a"}
Sep 10 13:36:32.089417 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:32.089388    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"]
Sep 10 13:36:32.091281 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:36:32.091249    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7173db0_0a67_497d_8e06_55bdb8a432e1.slice/crio-1cd40857ee66a2af47456914950c178eaa2177bda4c365b3d18e825c5fd1c922 WatchSource:0}: Error finding container 1cd40857ee66a2af47456914950c178eaa2177bda4c365b3d18e825c5fd1c922: Status 404 returned error can't find the container with id 1cd40857ee66a2af47456914950c178eaa2177bda4c365b3d18e825c5fd1c922
Sep 10 13:36:33.066349 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:33.066312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq" event={"ID":"b7173db0-0a67-497d-8e06-55bdb8a432e1","Type":"ContainerStarted","Data":"40d79968efecd30a0372664aa959d8b17d7481ffdcb4b3ae52e669e58c281a1d"}
Sep 10 13:36:33.066782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:33.066368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq" event={"ID":"b7173db0-0a67-497d-8e06-55bdb8a432e1","Type":"ContainerStarted","Data":"1cd40857ee66a2af47456914950c178eaa2177bda4c365b3d18e825c5fd1c922"}
Sep 10 13:36:33.066782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:33.066433    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:33.068036 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:33.068012    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd" event={"ID":"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1","Type":"ContainerStarted","Data":"c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33"}
Sep 10 13:36:33.068137 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:33.068097    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:33.089466 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:33.089410    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq" podStartSLOduration=2.08939303 podStartE2EDuration="2.08939303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:33.08728351 +0000 UTC m=+1098.104115473" watchObservedRunningTime="2026-09-10 13:36:33.08939303 +0000 UTC m=+1098.106225001"
Sep 10 13:36:33.113555 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:33.113497    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd" podStartSLOduration=2.798647746 podStartE2EDuration="3.113475822s" totalImagesPullingTime="314.828076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:33.112562761 +0000 UTC m=+1098.129394725" watchObservedRunningTime="2026-09-10 13:36:33.113475822 +0000 UTC m=+1098.130307795"
Sep 10 13:36:34.726472 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:34.726437    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:34.726869 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:34.726491    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:34.739391 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:34.739361    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:35.090701 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:35.090666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:36:39.079386 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:39.079357    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:36:44.076363 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:44.076331    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:36:51.193332 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.193300    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"]
Sep 10 13:36:51.193737 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.193526    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks" podUID="5cd3d037-f52b-4a5c-be6a-9eb1ad995425" containerName="maas-api" containerID="cri-o://1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254" gracePeriod=30
Sep 10 13:36:51.464364 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.464339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:51.601554 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.601510    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-metrics-certs\") pod \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") "
Sep 10 13:36:51.601758 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.601641    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx4rp\" (UniqueName: \"kubernetes.io/projected/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-kube-api-access-bx4rp\") pod \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") "
Sep 10 13:36:51.601758 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.601695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-openshift-service-ca\") pod \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") "
Sep 10 13:36:51.601758 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.601742    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-maas-api-tls\") pod \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\" (UID: \"5cd3d037-f52b-4a5c-be6a-9eb1ad995425\") "
Sep 10 13:36:51.602143 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.602109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5cd3d037-f52b-4a5c-be6a-9eb1ad995425" (UID: "5cd3d037-f52b-4a5c-be6a-9eb1ad995425"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:36:51.604169 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.604139    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5cd3d037-f52b-4a5c-be6a-9eb1ad995425" (UID: "5cd3d037-f52b-4a5c-be6a-9eb1ad995425"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:36:51.604433 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.604399    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-kube-api-access-bx4rp" (OuterVolumeSpecName: "kube-api-access-bx4rp") pod "5cd3d037-f52b-4a5c-be6a-9eb1ad995425" (UID: "5cd3d037-f52b-4a5c-be6a-9eb1ad995425"). InnerVolumeSpecName "kube-api-access-bx4rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:36:51.604540 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.604408    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5cd3d037-f52b-4a5c-be6a-9eb1ad995425" (UID: "5cd3d037-f52b-4a5c-be6a-9eb1ad995425"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:36:51.702729 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.702637    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:51.702729 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.702669    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:51.702729 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.702680    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx4rp\" (UniqueName: \"kubernetes.io/projected/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-kube-api-access-bx4rp\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:51.702729 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:51.702692    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd3d037-f52b-4a5c-be6a-9eb1ad995425-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:36:52.177308 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:52.177269    2578 generic.go:356] "Generic (PLEG): container finished" podID="5cd3d037-f52b-4a5c-be6a-9eb1ad995425" containerID="1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254" exitCode=0
Sep 10 13:36:52.177482 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:52.177319    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"
Sep 10 13:36:52.177482 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:52.177330    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks" event={"ID":"5cd3d037-f52b-4a5c-be6a-9eb1ad995425","Type":"ContainerDied","Data":"1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254"}
Sep 10 13:36:52.177482 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:52.177374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks" event={"ID":"5cd3d037-f52b-4a5c-be6a-9eb1ad995425","Type":"ContainerDied","Data":"dca2dd049fd4c7bc938f24392306655514dac4e7a5d38d698f31f83d8abaa44d"}
Sep 10 13:36:52.177482 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:52.177395    2578 scope.go:117] "RemoveContainer" containerID="1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254"
Sep 10 13:36:52.186834 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:52.186814    2578 scope.go:117] "RemoveContainer" containerID="1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254"
Sep 10 13:36:52.187128 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:36:52.187110    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254\": container with ID starting with 1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254 not found: ID does not exist" containerID="1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254"
Sep 10 13:36:52.187188 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:52.187135    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254"} err="failed to get container status \"1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254\": rpc error: code = NotFound desc = could not find container \"1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254\": container with ID starting with 1e5a6b507efaeec262ea854346ee6d22b7f368a21ba27d8251acc1270d112254 not found: ID does not exist"
Sep 10 13:36:52.201765 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:52.201717    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"]
Sep 10 13:36:52.211285 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:52.211253    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-95ae6288-5dd799ff87-gkwks"]
Sep 10 13:36:53.452948 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:53.452917    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cd3d037-f52b-4a5c-be6a-9eb1ad995425" path="/var/lib/kubelet/pods/5cd3d037-f52b-4a5c-be6a-9eb1ad995425/volumes"
Sep 10 13:36:57.723562 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.723525    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"]
Sep 10 13:36:57.723928 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.723916    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cd3d037-f52b-4a5c-be6a-9eb1ad995425" containerName="maas-api"
Sep 10 13:36:57.723972 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.723931    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cd3d037-f52b-4a5c-be6a-9eb1ad995425" containerName="maas-api"
Sep 10 13:36:57.724008 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.723996    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cd3d037-f52b-4a5c-be6a-9eb1ad995425" containerName="maas-api"
Sep 10 13:36:57.728898 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.728873    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.732391 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.732368    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2e7a3258-serving-cert\""
Sep 10 13:36:57.752900 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.752868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"]
Sep 10 13:36:57.856546 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.856512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-maas-api-tls\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.856716 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.856560    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4mz\" (UniqueName: \"kubernetes.io/projected/411811cd-2553-4386-9707-878e035d10b3-kube-api-access-lj4mz\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.856716 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.856595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-metrics-certs\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.856716 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.856665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411811cd-2553-4386-9707-878e035d10b3-openshift-service-ca\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.957402 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.957370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-maas-api-tls\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.957585 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.957423    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4mz\" (UniqueName: \"kubernetes.io/projected/411811cd-2553-4386-9707-878e035d10b3-kube-api-access-lj4mz\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.957585 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.957451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-metrics-certs\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.957585 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.957515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411811cd-2553-4386-9707-878e035d10b3-openshift-service-ca\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.958220 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.958162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411811cd-2553-4386-9707-878e035d10b3-openshift-service-ca\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.959842 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.959820    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-maas-api-tls\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.959973 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.959951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-metrics-certs\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:57.969149 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:57.969125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4mz\" (UniqueName: \"kubernetes.io/projected/411811cd-2553-4386-9707-878e035d10b3-kube-api-access-lj4mz\") pod \"maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:58.040365 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:58.040330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:58.318709 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:58.318683    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"]
Sep 10 13:36:58.321217 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:36:58.321173    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411811cd_2553_4386_9707_878e035d10b3.slice/crio-0ab6a8d2895aca53a10584755efe788e0223df0c577a5fdc756c506e2565b930 WatchSource:0}: Error finding container 0ab6a8d2895aca53a10584755efe788e0223df0c577a5fdc756c506e2565b930: Status 404 returned error can't find the container with id 0ab6a8d2895aca53a10584755efe788e0223df0c577a5fdc756c506e2565b930
Sep 10 13:36:59.217611 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:59.217558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw" event={"ID":"411811cd-2553-4386-9707-878e035d10b3","Type":"ContainerStarted","Data":"2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4"}
Sep 10 13:36:59.218109 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:59.217638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw" event={"ID":"411811cd-2553-4386-9707-878e035d10b3","Type":"ContainerStarted","Data":"0ab6a8d2895aca53a10584755efe788e0223df0c577a5fdc756c506e2565b930"}
Sep 10 13:36:59.218109 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:59.217796    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:36:59.243331 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:36:59.243281    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw" podStartSLOduration=1.887484351 podStartE2EDuration="2.243264207s" totalImagesPullingTime="355.779856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:36:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:36:59.241876913 +0000 UTC m=+1124.258708888" watchObservedRunningTime="2026-09-10 13:36:59.243264207 +0000 UTC m=+1124.260096180"
Sep 10 13:37:05.229700 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:05.229671    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:37:27.905372 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:27.905334    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"]
Sep 10 13:37:27.905925 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:27.905654    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw" podUID="411811cd-2553-4386-9707-878e035d10b3" containerName="maas-api" containerID="cri-o://2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4" gracePeriod=30
Sep 10 13:37:28.167152 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.167081    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:37:28.243116 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.243084    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-maas-api-tls\") pod \"411811cd-2553-4386-9707-878e035d10b3\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") "
Sep 10 13:37:28.243116 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.243117    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-metrics-certs\") pod \"411811cd-2553-4386-9707-878e035d10b3\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") "
Sep 10 13:37:28.243383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.243171    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj4mz\" (UniqueName: \"kubernetes.io/projected/411811cd-2553-4386-9707-878e035d10b3-kube-api-access-lj4mz\") pod \"411811cd-2553-4386-9707-878e035d10b3\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") "
Sep 10 13:37:28.243383 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.243283    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411811cd-2553-4386-9707-878e035d10b3-openshift-service-ca\") pod \"411811cd-2553-4386-9707-878e035d10b3\" (UID: \"411811cd-2553-4386-9707-878e035d10b3\") "
Sep 10 13:37:28.243727 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.243698    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411811cd-2553-4386-9707-878e035d10b3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "411811cd-2553-4386-9707-878e035d10b3" (UID: "411811cd-2553-4386-9707-878e035d10b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:37:28.245341 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.245311    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "411811cd-2553-4386-9707-878e035d10b3" (UID: "411811cd-2553-4386-9707-878e035d10b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:37:28.245440 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.245360    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "411811cd-2553-4386-9707-878e035d10b3" (UID: "411811cd-2553-4386-9707-878e035d10b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:37:28.245496 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.245431    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411811cd-2553-4386-9707-878e035d10b3-kube-api-access-lj4mz" (OuterVolumeSpecName: "kube-api-access-lj4mz") pod "411811cd-2553-4386-9707-878e035d10b3" (UID: "411811cd-2553-4386-9707-878e035d10b3"). InnerVolumeSpecName "kube-api-access-lj4mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:37:28.344471 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.344443    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:28.344471 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.344472    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/411811cd-2553-4386-9707-878e035d10b3-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:28.344737 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.344485    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4mz\" (UniqueName: \"kubernetes.io/projected/411811cd-2553-4386-9707-878e035d10b3-kube-api-access-lj4mz\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:28.344737 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.344503    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/411811cd-2553-4386-9707-878e035d10b3-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:28.380455 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.380420    2578 generic.go:356] "Generic (PLEG): container finished" podID="411811cd-2553-4386-9707-878e035d10b3" containerID="2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4" exitCode=0
Sep 10 13:37:28.380624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.380470    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"
Sep 10 13:37:28.380624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.380507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw" event={"ID":"411811cd-2553-4386-9707-878e035d10b3","Type":"ContainerDied","Data":"2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4"}
Sep 10 13:37:28.380624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.380594    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw" event={"ID":"411811cd-2553-4386-9707-878e035d10b3","Type":"ContainerDied","Data":"0ab6a8d2895aca53a10584755efe788e0223df0c577a5fdc756c506e2565b930"}
Sep 10 13:37:28.380746 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.380624    2578 scope.go:117] "RemoveContainer" containerID="2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4"
Sep 10 13:37:28.390928 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.390909    2578 scope.go:117] "RemoveContainer" containerID="2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4"
Sep 10 13:37:28.391263 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:37:28.391244    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4\": container with ID starting with 2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4 not found: ID does not exist" containerID="2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4"
Sep 10 13:37:28.391327 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.391272    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4"} err="failed to get container status \"2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4\": rpc error: code = NotFound desc = could not find container \"2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4\": container with ID starting with 2b54d9d0c33cf9e74f31f60d1e78dbf0b0d4c407375b9b1782e5f81771a6e4d4 not found: ID does not exist"
Sep 10 13:37:28.406295 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.406257    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"]
Sep 10 13:37:28.421344 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:28.419096    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2e7a3258-59d8d85797-kknbw"]
Sep 10 13:37:29.453020 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:29.452975    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="411811cd-2553-4386-9707-878e035d10b3" path="/var/lib/kubelet/pods/411811cd-2553-4386-9707-878e035d10b3/volumes"
Sep 10 13:37:37.112564 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.112533    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"]
Sep 10 13:37:37.113299 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.112740    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd" podUID="17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" containerName="maas-api" containerID="cri-o://c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33" gracePeriod=30
Sep 10 13:37:37.208479 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.208442    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"]
Sep 10 13:37:37.208757 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.208698    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq" podUID="b7173db0-0a67-497d-8e06-55bdb8a432e1" containerName="payload-pre-processing" containerID="cri-o://40d79968efecd30a0372664aa959d8b17d7481ffdcb4b3ae52e669e58c281a1d" gracePeriod=30
Sep 10 13:37:37.376586 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.376512    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:37:37.435091 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.435061    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7173db0-0a67-497d-8e06-55bdb8a432e1" containerID="40d79968efecd30a0372664aa959d8b17d7481ffdcb4b3ae52e669e58c281a1d" exitCode=0
Sep 10 13:37:37.435295 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.435141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq" event={"ID":"b7173db0-0a67-497d-8e06-55bdb8a432e1","Type":"ContainerDied","Data":"40d79968efecd30a0372664aa959d8b17d7481ffdcb4b3ae52e669e58c281a1d"}
Sep 10 13:37:37.436795 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.436773    2578 generic.go:356] "Generic (PLEG): container finished" podID="17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" containerID="c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33" exitCode=0
Sep 10 13:37:37.436917 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.436806    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd" event={"ID":"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1","Type":"ContainerDied","Data":"c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33"}
Sep 10 13:37:37.436917 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.436833    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"
Sep 10 13:37:37.436917 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.436868    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd" event={"ID":"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1","Type":"ContainerDied","Data":"ff8743a27fedc4c6cd7ae3f3f91f18763cc88aef69a7b3e283ed37019b213c2a"}
Sep 10 13:37:37.436917 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.436906    2578 scope.go:117] "RemoveContainer" containerID="c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33"
Sep 10 13:37:37.446979 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.446961    2578 scope.go:117] "RemoveContainer" containerID="c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33"
Sep 10 13:37:37.447294 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:37:37.447268    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33\": container with ID starting with c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33 not found: ID does not exist" containerID="c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33"
Sep 10 13:37:37.447395 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.447303    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33"} err="failed to get container status \"c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33\": rpc error: code = NotFound desc = could not find container \"c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33\": container with ID starting with c427364209e9eefbdb34b111c87b27327ed8e1d6dad39d87f7d131fc3ce4dd33 not found: ID does not exist"
Sep 10 13:37:37.480118 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.480095    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:37:37.531059 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.531035    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-openshift-service-ca\") pod \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") "
Sep 10 13:37:37.531274 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.531104    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-metrics-certs\") pod \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") "
Sep 10 13:37:37.531274 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.531124    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-maas-api-tls\") pod \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") "
Sep 10 13:37:37.531274 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.531168    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94wd2\" (UniqueName: \"kubernetes.io/projected/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-kube-api-access-94wd2\") pod \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\" (UID: \"17d7a5b0-ab91-4d67-a651-d1d7f24b33a1\") "
Sep 10 13:37:37.531498 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.531471    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" (UID: "17d7a5b0-ab91-4d67-a651-d1d7f24b33a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:37:37.533762 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.533732    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-kube-api-access-94wd2" (OuterVolumeSpecName: "kube-api-access-94wd2") pod "17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" (UID: "17d7a5b0-ab91-4d67-a651-d1d7f24b33a1"). InnerVolumeSpecName "kube-api-access-94wd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:37:37.533873 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.533785    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" (UID: "17d7a5b0-ab91-4d67-a651-d1d7f24b33a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:37:37.533873 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.533801    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" (UID: "17d7a5b0-ab91-4d67-a651-d1d7f24b33a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:37:37.632379 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.632299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7173db0-0a67-497d-8e06-55bdb8a432e1-plugins-config-volume\") pod \"b7173db0-0a67-497d-8e06-55bdb8a432e1\" (UID: \"b7173db0-0a67-497d-8e06-55bdb8a432e1\") "
Sep 10 13:37:37.632379 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.632352    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2x6v8\" (UniqueName: \"kubernetes.io/projected/b7173db0-0a67-497d-8e06-55bdb8a432e1-kube-api-access-2x6v8\") pod \"b7173db0-0a67-497d-8e06-55bdb8a432e1\" (UID: \"b7173db0-0a67-497d-8e06-55bdb8a432e1\") "
Sep 10 13:37:37.632710 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.632691    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:37.632749 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.632719    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:37.632749 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.632716    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7173db0-0a67-497d-8e06-55bdb8a432e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7173db0-0a67-497d-8e06-55bdb8a432e1" (UID: "b7173db0-0a67-497d-8e06-55bdb8a432e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:37:37.632749 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.632733    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:37.632749 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.632746    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94wd2\" (UniqueName: \"kubernetes.io/projected/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1-kube-api-access-94wd2\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:37.634442 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.634420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7173db0-0a67-497d-8e06-55bdb8a432e1-kube-api-access-2x6v8" (OuterVolumeSpecName: "kube-api-access-2x6v8") pod "b7173db0-0a67-497d-8e06-55bdb8a432e1" (UID: "b7173db0-0a67-497d-8e06-55bdb8a432e1"). InnerVolumeSpecName "kube-api-access-2x6v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:37:37.733849 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.733775    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7173db0-0a67-497d-8e06-55bdb8a432e1-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:37.733849 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.733802    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x6v8\" (UniqueName: \"kubernetes.io/projected/b7173db0-0a67-497d-8e06-55bdb8a432e1-kube-api-access-2x6v8\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:37:37.761266 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.761230    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"]
Sep 10 13:37:37.767763 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:37.767740    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3c32b1-86d4ffbfdc-cnkrd"]
Sep 10 13:37:38.449387 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:38.449299    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"
Sep 10 13:37:38.449387 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:38.449315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq" event={"ID":"b7173db0-0a67-497d-8e06-55bdb8a432e1","Type":"ContainerDied","Data":"1cd40857ee66a2af47456914950c178eaa2177bda4c365b3d18e825c5fd1c922"}
Sep 10 13:37:38.449873 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:38.449387    2578 scope.go:117] "RemoveContainer" containerID="40d79968efecd30a0372664aa959d8b17d7481ffdcb4b3ae52e669e58c281a1d"
Sep 10 13:37:38.478827 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:38.478801    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"]
Sep 10 13:37:38.480747 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:38.480728    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-3c32b1-84f9dcd64b-wdvdq"]
Sep 10 13:37:39.451887 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:39.451851    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" path="/var/lib/kubelet/pods/17d7a5b0-ab91-4d67-a651-d1d7f24b33a1/volumes"
Sep 10 13:37:39.452399 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:37:39.452380    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7173db0-0a67-497d-8e06-55bdb8a432e1" path="/var/lib/kubelet/pods/b7173db0-0a67-497d-8e06-55bdb8a432e1/volumes"
Sep 10 13:38:35.682956 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:35.682925    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"]
Sep 10 13:38:47.555299 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.555267    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"]
Sep 10 13:38:47.555681 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.555470    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj" podUID="4d81df7b-efe0-407e-b0f9-0283056e7499" containerName="maas-api" containerID="cri-o://91597e6ce29538b9cb37a0bade85fab31987faca0304445f4b5d07cb41991dc3" gracePeriod=30
Sep 10 13:38:47.794447 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.794420    2578 generic.go:356] "Generic (PLEG): container finished" podID="4d81df7b-efe0-407e-b0f9-0283056e7499" containerID="91597e6ce29538b9cb37a0bade85fab31987faca0304445f4b5d07cb41991dc3" exitCode=0
Sep 10 13:38:47.794591 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.794463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj" event={"ID":"4d81df7b-efe0-407e-b0f9-0283056e7499","Type":"ContainerDied","Data":"91597e6ce29538b9cb37a0bade85fab31987faca0304445f4b5d07cb41991dc3"}
Sep 10 13:38:47.813462 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.813404    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:38:47.870991 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.870955    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d81df7b-efe0-407e-b0f9-0283056e7499-openshift-service-ca\") pod \"4d81df7b-efe0-407e-b0f9-0283056e7499\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") "
Sep 10 13:38:47.870991 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.870997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-metrics-certs\") pod \"4d81df7b-efe0-407e-b0f9-0283056e7499\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") "
Sep 10 13:38:47.871183 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.871037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg7ps\" (UniqueName: \"kubernetes.io/projected/4d81df7b-efe0-407e-b0f9-0283056e7499-kube-api-access-kg7ps\") pod \"4d81df7b-efe0-407e-b0f9-0283056e7499\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") "
Sep 10 13:38:47.871263 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.871216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-maas-api-tls\") pod \"4d81df7b-efe0-407e-b0f9-0283056e7499\" (UID: \"4d81df7b-efe0-407e-b0f9-0283056e7499\") "
Sep 10 13:38:47.871450 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.871419    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d81df7b-efe0-407e-b0f9-0283056e7499-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d81df7b-efe0-407e-b0f9-0283056e7499" (UID: "4d81df7b-efe0-407e-b0f9-0283056e7499"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:38:47.871587 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.871555    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d81df7b-efe0-407e-b0f9-0283056e7499-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:47.873187 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.873160    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d81df7b-efe0-407e-b0f9-0283056e7499-kube-api-access-kg7ps" (OuterVolumeSpecName: "kube-api-access-kg7ps") pod "4d81df7b-efe0-407e-b0f9-0283056e7499" (UID: "4d81df7b-efe0-407e-b0f9-0283056e7499"). InnerVolumeSpecName "kube-api-access-kg7ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:38:47.873301 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.873180    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d81df7b-efe0-407e-b0f9-0283056e7499" (UID: "4d81df7b-efe0-407e-b0f9-0283056e7499"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:38:47.873301 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.873278    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4d81df7b-efe0-407e-b0f9-0283056e7499" (UID: "4d81df7b-efe0-407e-b0f9-0283056e7499"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:38:47.972397 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.972367    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:47.972397 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.972393    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg7ps\" (UniqueName: \"kubernetes.io/projected/4d81df7b-efe0-407e-b0f9-0283056e7499-kube-api-access-kg7ps\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:47.972397 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:47.972403    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d81df7b-efe0-407e-b0f9-0283056e7499-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:48.800633 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:48.800604    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"
Sep 10 13:38:48.801031 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:48.800601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj" event={"ID":"4d81df7b-efe0-407e-b0f9-0283056e7499","Type":"ContainerDied","Data":"4e4810a01a8f4953ffc1ce1f261ba845b20e6c461b2718264161206344f40c05"}
Sep 10 13:38:48.801031 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:48.800753    2578 scope.go:117] "RemoveContainer" containerID="91597e6ce29538b9cb37a0bade85fab31987faca0304445f4b5d07cb41991dc3"
Sep 10 13:38:48.828809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:48.828777    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"]
Sep 10 13:38:48.831225 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:48.831187    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc03be-5fc5cddb69-lblkj"]
Sep 10 13:38:49.452090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:49.452055    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d81df7b-efe0-407e-b0f9-0283056e7499" path="/var/lib/kubelet/pods/4d81df7b-efe0-407e-b0f9-0283056e7499/volumes"
Sep 10 13:38:50.702651 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:50.702608    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp" podUID="a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" containerName="main" containerID="cri-o://d2ff2de88acb3fc9b9f4784e4d379092f45356e8fc7b52c43c0f2a3125bb5ad5" gracePeriod=45
Sep 10 13:38:50.816884 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:50.816856    2578 generic.go:356] "Generic (PLEG): container finished" podID="a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" containerID="d2ff2de88acb3fc9b9f4784e4d379092f45356e8fc7b52c43c0f2a3125bb5ad5" exitCode=0
Sep 10 13:38:50.817013 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:50.816927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp" event={"ID":"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d","Type":"ContainerDied","Data":"d2ff2de88acb3fc9b9f4784e4d379092f45356e8fc7b52c43c0f2a3125bb5ad5"}
Sep 10 13:38:50.950361 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:50.950340    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:38:51.002909 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.002866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tmp-dir\") pod \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") "
Sep 10 13:38:51.003119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.002927    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpxjt\" (UniqueName: \"kubernetes.io/projected/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-kube-api-access-wpxjt\") pod \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") "
Sep 10 13:38:51.003119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.002956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-home\") pod \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") "
Sep 10 13:38:51.003119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.002990    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-model-cache\") pod \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") "
Sep 10 13:38:51.003119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.003018    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-dshm\") pod \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") "
Sep 10 13:38:51.003119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.003043    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tls-certs\") pod \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\" (UID: \"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d\") "
Sep 10 13:38:51.003421 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.003217    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" (UID: "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:38:51.003421 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.003381    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tmp-dir\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:51.003528 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.003482    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" (UID: "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:38:51.003585 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.003546    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-home" (OuterVolumeSpecName: "home") pod "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" (UID: "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:38:51.005881 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.005846    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-dshm" (OuterVolumeSpecName: "dshm") pod "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" (UID: "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:38:51.006010 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.005905    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-kube-api-access-wpxjt" (OuterVolumeSpecName: "kube-api-access-wpxjt") pod "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" (UID: "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d"). InnerVolumeSpecName "kube-api-access-wpxjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:38:51.006010 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.005929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" (UID: "a71c4bf3-f3a5-47f2-9a4f-59ee869c282d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:38:51.103896 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.103861    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-model-cache\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:51.103896 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.103897    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-dshm\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:51.104107 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.103908    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-tls-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:51.104107 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.103921    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxjt\" (UniqueName: \"kubernetes.io/projected/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-kube-api-access-wpxjt\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:51.104107 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.103933    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d-home\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:38:51.822933 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.822899    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"
Sep 10 13:38:51.823356 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.822898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp" event={"ID":"a71c4bf3-f3a5-47f2-9a4f-59ee869c282d","Type":"ContainerDied","Data":"b0e34cb66ceed703725bfe4e4eda7c4a876d629172a5308adb4aaae3891ba8cd"}
Sep 10 13:38:51.823356 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.823025    2578 scope.go:117] "RemoveContainer" containerID="d2ff2de88acb3fc9b9f4784e4d379092f45356e8fc7b52c43c0f2a3125bb5ad5"
Sep 10 13:38:51.849335 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.849306    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"]
Sep 10 13:38:51.854265 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:51.854243    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bc03be/test-model-bc03be-kserve-5cbcb49c4c-sf6zp"]
Sep 10 13:38:53.452346 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:38:53.452306    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" path="/var/lib/kubelet/pods/a71c4bf3-f3a5-47f2-9a4f-59ee869c282d/volumes"
Sep 10 13:39:08.927419 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:08.927328    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"]
Sep 10 13:39:08.927849 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:08.927648    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk" podUID="325c1350-25a8-4bc4-b791-0858b325e0c0" containerName="maas-api" containerID="cri-o://0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92" gracePeriod=30
Sep 10 13:39:09.066563 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.066532    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"]
Sep 10 13:39:09.066778 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.066740    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w" podUID="fdc038d6-c745-46a0-bed1-ddca6e90470a" containerName="payload-processing" containerID="cri-o://62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9" gracePeriod=30
Sep 10 13:39:09.333003 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.332977    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:39:09.386308 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.386281    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:39:09.473684 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.473605    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd8zn\" (UniqueName: \"kubernetes.io/projected/fdc038d6-c745-46a0-bed1-ddca6e90470a-kube-api-access-fd8zn\") pod \"fdc038d6-c745-46a0-bed1-ddca6e90470a\" (UID: \"fdc038d6-c745-46a0-bed1-ddca6e90470a\") "
Sep 10 13:39:09.473684 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.473634    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc038d6-c745-46a0-bed1-ddca6e90470a-plugins-config-volume\") pod \"fdc038d6-c745-46a0-bed1-ddca6e90470a\" (UID: \"fdc038d6-c745-46a0-bed1-ddca6e90470a\") "
Sep 10 13:39:09.474039 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.474015    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc038d6-c745-46a0-bed1-ddca6e90470a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdc038d6-c745-46a0-bed1-ddca6e90470a" (UID: "fdc038d6-c745-46a0-bed1-ddca6e90470a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:39:09.475577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.475558    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc038d6-c745-46a0-bed1-ddca6e90470a-kube-api-access-fd8zn" (OuterVolumeSpecName: "kube-api-access-fd8zn") pod "fdc038d6-c745-46a0-bed1-ddca6e90470a" (UID: "fdc038d6-c745-46a0-bed1-ddca6e90470a"). InnerVolumeSpecName "kube-api-access-fd8zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:39:09.574335 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.574299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2c66\" (UniqueName: \"kubernetes.io/projected/325c1350-25a8-4bc4-b791-0858b325e0c0-kube-api-access-p2c66\") pod \"325c1350-25a8-4bc4-b791-0858b325e0c0\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") "
Sep 10 13:39:09.574529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.574347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-maas-api-tls\") pod \"325c1350-25a8-4bc4-b791-0858b325e0c0\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") "
Sep 10 13:39:09.574529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.574375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-metrics-certs\") pod \"325c1350-25a8-4bc4-b791-0858b325e0c0\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") "
Sep 10 13:39:09.574529 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.574403    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325c1350-25a8-4bc4-b791-0858b325e0c0-openshift-service-ca\") pod \"325c1350-25a8-4bc4-b791-0858b325e0c0\" (UID: \"325c1350-25a8-4bc4-b791-0858b325e0c0\") "
Sep 10 13:39:09.574715 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.574567    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd8zn\" (UniqueName: \"kubernetes.io/projected/fdc038d6-c745-46a0-bed1-ddca6e90470a-kube-api-access-fd8zn\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:39:09.574715 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.574576    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc038d6-c745-46a0-bed1-ddca6e90470a-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:39:09.574848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.574817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325c1350-25a8-4bc4-b791-0858b325e0c0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "325c1350-25a8-4bc4-b791-0858b325e0c0" (UID: "325c1350-25a8-4bc4-b791-0858b325e0c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:39:09.576500 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.576472    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "325c1350-25a8-4bc4-b791-0858b325e0c0" (UID: "325c1350-25a8-4bc4-b791-0858b325e0c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:39:09.576592 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.576466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325c1350-25a8-4bc4-b791-0858b325e0c0-kube-api-access-p2c66" (OuterVolumeSpecName: "kube-api-access-p2c66") pod "325c1350-25a8-4bc4-b791-0858b325e0c0" (UID: "325c1350-25a8-4bc4-b791-0858b325e0c0"). InnerVolumeSpecName "kube-api-access-p2c66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:39:09.576592 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.576500    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "325c1350-25a8-4bc4-b791-0858b325e0c0" (UID: "325c1350-25a8-4bc4-b791-0858b325e0c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:39:09.675748 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.675706    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:39:09.675748 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.675742    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/325c1350-25a8-4bc4-b791-0858b325e0c0-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:39:09.675748 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.675757    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2c66\" (UniqueName: \"kubernetes.io/projected/325c1350-25a8-4bc4-b791-0858b325e0c0-kube-api-access-p2c66\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:39:09.675985 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.675770    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/325c1350-25a8-4bc4-b791-0858b325e0c0-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:39:09.914578 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.914543    2578 generic.go:356] "Generic (PLEG): container finished" podID="325c1350-25a8-4bc4-b791-0858b325e0c0" containerID="0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92" exitCode=0
Sep 10 13:39:09.914752 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.914588    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"
Sep 10 13:39:09.914752 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.914625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk" event={"ID":"325c1350-25a8-4bc4-b791-0858b325e0c0","Type":"ContainerDied","Data":"0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92"}
Sep 10 13:39:09.914752 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.914676    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk" event={"ID":"325c1350-25a8-4bc4-b791-0858b325e0c0","Type":"ContainerDied","Data":"4e66106fcff0e63fdfe70388a6ff22ccb3a7312d6f04cbd62e88c455be4ae03f"}
Sep 10 13:39:09.914752 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.914704    2578 scope.go:117] "RemoveContainer" containerID="0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92"
Sep 10 13:39:09.916212 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.916170    2578 generic.go:356] "Generic (PLEG): container finished" podID="fdc038d6-c745-46a0-bed1-ddca6e90470a" containerID="62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9" exitCode=0
Sep 10 13:39:09.916399 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.916237    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"
Sep 10 13:39:09.916399 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.916231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w" event={"ID":"fdc038d6-c745-46a0-bed1-ddca6e90470a","Type":"ContainerDied","Data":"62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9"}
Sep 10 13:39:09.916399 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.916352    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w" event={"ID":"fdc038d6-c745-46a0-bed1-ddca6e90470a","Type":"ContainerDied","Data":"71c679e24ec067c3f69b8c7d59c6254d0dd34ac20941d6d05548806afe88ec20"}
Sep 10 13:39:09.926634 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.926611    2578 scope.go:117] "RemoveContainer" containerID="0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92"
Sep 10 13:39:09.927135 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:39:09.926972    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92\": container with ID starting with 0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92 not found: ID does not exist" containerID="0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92"
Sep 10 13:39:09.927135 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.927001    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92"} err="failed to get container status \"0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92\": rpc error: code = NotFound desc = could not find container \"0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92\": container with ID starting with 0b11e62aa5f224b7a666aa170e58b6015177400c90baa21b91895d1cbdca0e92 not found: ID does not exist"
Sep 10 13:39:09.927135 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.927020    2578 scope.go:117] "RemoveContainer" containerID="62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9"
Sep 10 13:39:09.935462 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.935310    2578 scope.go:117] "RemoveContainer" containerID="62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9"
Sep 10 13:39:09.935694 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:39:09.935562    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9\": container with ID starting with 62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9 not found: ID does not exist" containerID="62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9"
Sep 10 13:39:09.935694 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.935582    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9"} err="failed to get container status \"62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9\": rpc error: code = NotFound desc = could not find container \"62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9\": container with ID starting with 62742e47d5e9ef14d96261a5ea0b55f72811dbac5d378673226fd75fc6ef68c9 not found: ID does not exist"
Sep 10 13:39:09.949319 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.947323    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"]
Sep 10 13:39:09.955143 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.955110    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d6159c-7d9cf57f97-zv9vk"]
Sep 10 13:39:09.969541 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.969515    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"]
Sep 10 13:39:09.976411 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:09.976385    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d6159c-65fd8cdfbc-8562w"]
Sep 10 13:39:11.451846 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:11.451810    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="325c1350-25a8-4bc4-b791-0858b325e0c0" path="/var/lib/kubelet/pods/325c1350-25a8-4bc4-b791-0858b325e0c0/volumes"
Sep 10 13:39:11.452218 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:11.452163    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc038d6-c745-46a0-bed1-ddca6e90470a" path="/var/lib/kubelet/pods/fdc038d6-c745-46a0-bed1-ddca6e90470a/volumes"
Sep 10 13:39:19.168512 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.168482    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"]
Sep 10 13:39:19.169018 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.168999    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc038d6-c745-46a0-bed1-ddca6e90470a" containerName="payload-processing"
Sep 10 13:39:19.169099 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169021    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc038d6-c745-46a0-bed1-ddca6e90470a" containerName="payload-processing"
Sep 10 13:39:19.169099 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169037    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d81df7b-efe0-407e-b0f9-0283056e7499" containerName="maas-api"
Sep 10 13:39:19.169099 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169048    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d81df7b-efe0-407e-b0f9-0283056e7499" containerName="maas-api"
Sep 10 13:39:19.169099 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169060    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" containerName="main"
Sep 10 13:39:19.169099 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169069    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" containerName="main"
Sep 10 13:39:19.169099 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169089    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="411811cd-2553-4386-9707-878e035d10b3" containerName="maas-api"
Sep 10 13:39:19.169099 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169097    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="411811cd-2553-4386-9707-878e035d10b3" containerName="maas-api"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169109    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7173db0-0a67-497d-8e06-55bdb8a432e1" containerName="payload-pre-processing"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169118    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7173db0-0a67-497d-8e06-55bdb8a432e1" containerName="payload-pre-processing"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169129    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" containerName="maas-api"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169138    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" containerName="maas-api"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169157    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325c1350-25a8-4bc4-b791-0858b325e0c0" containerName="maas-api"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169164    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="325c1350-25a8-4bc4-b791-0858b325e0c0" containerName="maas-api"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169274    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="325c1350-25a8-4bc4-b791-0858b325e0c0" containerName="maas-api"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169289    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d81df7b-efe0-407e-b0f9-0283056e7499" containerName="maas-api"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169298    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="411811cd-2553-4386-9707-878e035d10b3" containerName="maas-api"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169310    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17d7a5b0-ab91-4d67-a651-d1d7f24b33a1" containerName="maas-api"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169318    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71c4bf3-f3a5-47f2-9a4f-59ee869c282d" containerName="main"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169325    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7173db0-0a67-497d-8e06-55bdb8a432e1" containerName="payload-pre-processing"
Sep 10 13:39:19.169468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.169335    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc038d6-c745-46a0-bed1-ddca6e90470a" containerName="payload-processing"
Sep 10 13:39:19.174079 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.174058    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.177029 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.177001    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-839f01ff-serving-cert\""
Sep 10 13:39:19.187941 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.187914    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"]
Sep 10 13:39:19.258052 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.258014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-maas-api-tls\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.258275 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.258058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ddfe99-db81-4b6e-8848-409cdd16e19d-openshift-service-ca\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.258275 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.258098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f42t\" (UniqueName: \"kubernetes.io/projected/92ddfe99-db81-4b6e-8848-409cdd16e19d-kube-api-access-5f42t\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.258275 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.258127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-metrics-certs\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.359672 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.359638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ddfe99-db81-4b6e-8848-409cdd16e19d-openshift-service-ca\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.359672 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.359683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f42t\" (UniqueName: \"kubernetes.io/projected/92ddfe99-db81-4b6e-8848-409cdd16e19d-kube-api-access-5f42t\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.359895 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.359702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-metrics-certs\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.359895 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.359770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-maas-api-tls\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.360412 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.360385    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ddfe99-db81-4b6e-8848-409cdd16e19d-openshift-service-ca\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.362240 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.362214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-maas-api-tls\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.362353 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.362266    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-metrics-certs\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.374015 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.373992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f42t\" (UniqueName: \"kubernetes.io/projected/92ddfe99-db81-4b6e-8848-409cdd16e19d-kube-api-access-5f42t\") pod \"maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.483809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.483691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:19.773460 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.773433    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"]
Sep 10 13:39:19.775874 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:39:19.775846    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ddfe99_db81_4b6e_8848_409cdd16e19d.slice/crio-d6af352315e019d57af7f2c8e21e47961a16a4a9598bda511a940bcbaf4521b3 WatchSource:0}: Error finding container d6af352315e019d57af7f2c8e21e47961a16a4a9598bda511a940bcbaf4521b3: Status 404 returned error can't find the container with id d6af352315e019d57af7f2c8e21e47961a16a4a9598bda511a940bcbaf4521b3
Sep 10 13:39:19.777284 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.777267    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:39:19.970517 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:19.970479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz" event={"ID":"92ddfe99-db81-4b6e-8848-409cdd16e19d","Type":"ContainerStarted","Data":"d6af352315e019d57af7f2c8e21e47961a16a4a9598bda511a940bcbaf4521b3"}
Sep 10 13:39:20.339263 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.339233    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"]
Sep 10 13:39:20.343885 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.343867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:20.351983 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.351961    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-839f01ff\""
Sep 10 13:39:20.358090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.358072    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-839f01ff-dockercfg-k6rkd\""
Sep 10 13:39:20.369453 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.369429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c0584c-d13e-4e12-b0f7-df7cdabdd156-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42\" (UID: \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:20.369539 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.369455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjg8\" (UniqueName: \"kubernetes.io/projected/38c0584c-d13e-4e12-b0f7-df7cdabdd156-kube-api-access-wtjg8\") pod \"payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42\" (UID: \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:20.382898 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.382869    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"]
Sep 10 13:39:20.439622 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.439583    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"]
Sep 10 13:39:20.443106 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.443085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:20.470622 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.470586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqkmb\" (UniqueName: \"kubernetes.io/projected/cbcda5e1-fbf9-4902-941f-defd28ec25cc-kube-api-access-gqkmb\") pod \"payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn\" (UID: \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\") " pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:20.470814 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.470693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcda5e1-fbf9-4902-941f-defd28ec25cc-plugins-config-volume\") pod \"payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn\" (UID: \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\") " pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:20.470814 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.470726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c0584c-d13e-4e12-b0f7-df7cdabdd156-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42\" (UID: \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:20.470814 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.470773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjg8\" (UniqueName: \"kubernetes.io/projected/38c0584c-d13e-4e12-b0f7-df7cdabdd156-kube-api-access-wtjg8\") pod \"payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42\" (UID: \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:20.471415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.471394    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c0584c-d13e-4e12-b0f7-df7cdabdd156-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42\" (UID: \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:20.483288 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.483251    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjg8\" (UniqueName: \"kubernetes.io/projected/38c0584c-d13e-4e12-b0f7-df7cdabdd156-kube-api-access-wtjg8\") pod \"payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42\" (UID: \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:20.507432 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.507256    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"]
Sep 10 13:39:20.572266 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.572222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcda5e1-fbf9-4902-941f-defd28ec25cc-plugins-config-volume\") pod \"payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn\" (UID: \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\") " pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:20.572398 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.572307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqkmb\" (UniqueName: \"kubernetes.io/projected/cbcda5e1-fbf9-4902-941f-defd28ec25cc-kube-api-access-gqkmb\") pod \"payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn\" (UID: \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\") " pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:20.573086 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.573062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcda5e1-fbf9-4902-941f-defd28ec25cc-plugins-config-volume\") pod \"payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn\" (UID: \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\") " pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:20.597295 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.597221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqkmb\" (UniqueName: \"kubernetes.io/projected/cbcda5e1-fbf9-4902-941f-defd28ec25cc-kube-api-access-gqkmb\") pod \"payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn\" (UID: \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\") " pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:20.654684 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.654652    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:20.753796 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.753760    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:20.960616 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.960583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"]
Sep 10 13:39:20.965046 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:39:20.965021    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c0584c_d13e_4e12_b0f7_df7cdabdd156.slice/crio-370cfe9b82f871b932173bdb03f35fe211618f50b05844017bdbd7b3b7b63279 WatchSource:0}: Error finding container 370cfe9b82f871b932173bdb03f35fe211618f50b05844017bdbd7b3b7b63279: Status 404 returned error can't find the container with id 370cfe9b82f871b932173bdb03f35fe211618f50b05844017bdbd7b3b7b63279
Sep 10 13:39:20.977065 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.977035    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz" event={"ID":"92ddfe99-db81-4b6e-8848-409cdd16e19d","Type":"ContainerStarted","Data":"5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b"}
Sep 10 13:39:20.977188 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.977102    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:20.978367 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:20.978343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42" event={"ID":"38c0584c-d13e-4e12-b0f7-df7cdabdd156","Type":"ContainerStarted","Data":"370cfe9b82f871b932173bdb03f35fe211618f50b05844017bdbd7b3b7b63279"}
Sep 10 13:39:21.019868 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:21.019820    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz" podStartSLOduration=1.289466343 podStartE2EDuration="2.019803335s" totalImagesPullingTime="730.336992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:21.018029293 +0000 UTC m=+1266.034861265" watchObservedRunningTime="2026-09-10 13:39:21.019803335 +0000 UTC m=+1266.036635307"
Sep 10 13:39:21.046213 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:21.045780    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"]
Sep 10 13:39:21.049776 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:39:21.049751    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbcda5e1_fbf9_4902_941f_defd28ec25cc.slice/crio-41f8aed5c8ae641c8155f68120d9bd58ffad3829c0b5a786555e7917b3575900 WatchSource:0}: Error finding container 41f8aed5c8ae641c8155f68120d9bd58ffad3829c0b5a786555e7917b3575900: Status 404 returned error can't find the container with id 41f8aed5c8ae641c8155f68120d9bd58ffad3829c0b5a786555e7917b3575900
Sep 10 13:39:21.985112 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:21.985075    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42" event={"ID":"38c0584c-d13e-4e12-b0f7-df7cdabdd156","Type":"ContainerStarted","Data":"8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a"}
Sep 10 13:39:21.987365 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:21.985145    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:21.987365 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:21.986714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn" event={"ID":"cbcda5e1-fbf9-4902-941f-defd28ec25cc","Type":"ContainerStarted","Data":"4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a"}
Sep 10 13:39:21.987365 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:21.986779    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:21.987365 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:21.986798    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn" event={"ID":"cbcda5e1-fbf9-4902-941f-defd28ec25cc","Type":"ContainerStarted","Data":"41f8aed5c8ae641c8155f68120d9bd58ffad3829c0b5a786555e7917b3575900"}
Sep 10 13:39:22.014936 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:22.014884    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42" podStartSLOduration=2.014870083 podStartE2EDuration="2.014870083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:22.011951678 +0000 UTC m=+1267.028783650" watchObservedRunningTime="2026-09-10 13:39:22.014870083 +0000 UTC m=+1267.031702054"
Sep 10 13:39:22.042337 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:22.042279    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn" podStartSLOduration=2.042260761 podStartE2EDuration="2.042260761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:22.040378043 +0000 UTC m=+1267.057210017" watchObservedRunningTime="2026-09-10 13:39:22.042260761 +0000 UTC m=+1267.059092733"
Sep 10 13:39:24.172034 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.171999    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"]
Sep 10 13:39:24.175444 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.175422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.178082 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.178063    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-767dce-serving-cert\""
Sep 10 13:39:24.193755 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.193729    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"]
Sep 10 13:39:24.203679 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.203646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f439fcff-d8e2-419f-8a2c-b10b1a513422-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.203878 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.203740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgpj7\" (UniqueName: \"kubernetes.io/projected/f439fcff-d8e2-419f-8a2c-b10b1a513422-kube-api-access-jgpj7\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.203878 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.203844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.204005 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.203897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.305109 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.305071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgpj7\" (UniqueName: \"kubernetes.io/projected/f439fcff-d8e2-419f-8a2c-b10b1a513422-kube-api-access-jgpj7\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.305109 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.305115    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.305479 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.305142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.305479 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:39:24.305265    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-serving-cert: secret "maas-api-e2e-shared-a-w3-767dce-serving-cert" not found
Sep 10 13:39:24.305479 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.305319    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f439fcff-d8e2-419f-8a2c-b10b1a513422-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.305479 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:39:24.305329    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls podName:f439fcff-d8e2-419f-8a2c-b10b1a513422 nodeName:}" failed. No retries permitted until 2026-09-10 13:39:24.805309115 +0000 UTC m=+1269.822141088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls") pod "maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc" (UID: "f439fcff-d8e2-419f-8a2c-b10b1a513422") : secret "maas-api-e2e-shared-a-w3-767dce-serving-cert" not found
Sep 10 13:39:24.305936 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.305916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f439fcff-d8e2-419f-8a2c-b10b1a513422-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.307570 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.307549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.317016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.316994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgpj7\" (UniqueName: \"kubernetes.io/projected/f439fcff-d8e2-419f-8a2c-b10b1a513422-kube-api-access-jgpj7\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.809907 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:24.809864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:24.810107 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:39:24.810021    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-serving-cert: secret "maas-api-e2e-shared-a-w3-767dce-serving-cert" not found
Sep 10 13:39:24.810107 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:39:24.810086    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls podName:f439fcff-d8e2-419f-8a2c-b10b1a513422 nodeName:}" failed. No retries permitted until 2026-09-10 13:39:25.810070488 +0000 UTC m=+1270.826902442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls") pod "maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc" (UID: "f439fcff-d8e2-419f-8a2c-b10b1a513422") : secret "maas-api-e2e-shared-a-w3-767dce-serving-cert" not found
Sep 10 13:39:25.821275 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:25.821233    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:25.821734 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:39:25.821407    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-serving-cert: secret "maas-api-e2e-shared-a-w3-767dce-serving-cert" not found
Sep 10 13:39:25.821734 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:39:25.821510    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls podName:f439fcff-d8e2-419f-8a2c-b10b1a513422 nodeName:}" failed. No retries permitted until 2026-09-10 13:39:27.82149257 +0000 UTC m=+1272.838324521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls") pod "maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc" (UID: "f439fcff-d8e2-419f-8a2c-b10b1a513422") : secret "maas-api-e2e-shared-a-w3-767dce-serving-cert" not found
Sep 10 13:39:26.992498 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:26.992469    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:39:27.842181 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:27.842141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:27.844533 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:27.844504    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:28.087419 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:28.087378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:28.369531 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:28.369505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"]
Sep 10 13:39:28.371517 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:39:28.371487    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf439fcff_d8e2_419f_8a2c_b10b1a513422.slice/crio-792bee1982bbd895fe4a0696e00fc1dcad51a1cca6b13fc77a850fe2d820f865 WatchSource:0}: Error finding container 792bee1982bbd895fe4a0696e00fc1dcad51a1cca6b13fc77a850fe2d820f865: Status 404 returned error can't find the container with id 792bee1982bbd895fe4a0696e00fc1dcad51a1cca6b13fc77a850fe2d820f865
Sep 10 13:39:29.026921 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:29.026884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc" event={"ID":"f439fcff-d8e2-419f-8a2c-b10b1a513422","Type":"ContainerStarted","Data":"c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c"}
Sep 10 13:39:29.027108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:29.026948    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc" event={"ID":"f439fcff-d8e2-419f-8a2c-b10b1a513422","Type":"ContainerStarted","Data":"792bee1982bbd895fe4a0696e00fc1dcad51a1cca6b13fc77a850fe2d820f865"}
Sep 10 13:39:29.027108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:29.027004    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:29.048859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:29.048810    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc" podStartSLOduration=4.466206463 podStartE2EDuration="5.048794686s" totalImagesPullingTime="582.588223ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:29.046677476 +0000 UTC m=+1274.063509447" watchObservedRunningTime="2026-09-10 13:39:29.048794686 +0000 UTC m=+1274.065626657"
Sep 10 13:39:32.994613 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:32.994577    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:39:32.994995 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:32.994752    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:39:35.036455 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:35.036429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:39:53.626923 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.626886    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"]
Sep 10 13:39:53.630583 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.630561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.637157 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.637130    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-89be92-serving-cert\""
Sep 10 13:39:53.658251 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.658217    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"]
Sep 10 13:39:53.778820 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.778786    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.779016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.778832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.779016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.778865    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e497323-636b-46d2-8203-34c432db324c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.779016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.778932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndk7\" (UniqueName: \"kubernetes.io/projected/9e497323-636b-46d2-8203-34c432db324c-kube-api-access-dndk7\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.880255 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.880135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.880255 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.880187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e497323-636b-46d2-8203-34c432db324c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.880255 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.880246    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndk7\" (UniqueName: \"kubernetes.io/projected/9e497323-636b-46d2-8203-34c432db324c-kube-api-access-dndk7\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.880554 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.880304    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.880930 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.880906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e497323-636b-46d2-8203-34c432db324c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.882777 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.882741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.882892 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.882795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.896365 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.896340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndk7\" (UniqueName: \"kubernetes.io/projected/9e497323-636b-46d2-8203-34c432db324c-kube-api-access-dndk7\") pod \"maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:53.943263 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:53.943226    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:54.225269 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:54.225238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"]
Sep 10 13:39:54.227583 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:39:54.227552    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e497323_636b_46d2_8203_34c432db324c.slice/crio-a4ff3dc46e4fd296cfc877029397226f309df4dbc03e3ae559d07b96fcbe3d41 WatchSource:0}: Error finding container a4ff3dc46e4fd296cfc877029397226f309df4dbc03e3ae559d07b96fcbe3d41: Status 404 returned error can't find the container with id a4ff3dc46e4fd296cfc877029397226f309df4dbc03e3ae559d07b96fcbe3d41
Sep 10 13:39:55.160832 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:55.160793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc" event={"ID":"9e497323-636b-46d2-8203-34c432db324c","Type":"ContainerStarted","Data":"5af82144c9e1100ca986459164e0b1afeb9dfe67717159fe0614e14fc6a81c90"}
Sep 10 13:39:55.161313 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:55.160850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc" event={"ID":"9e497323-636b-46d2-8203-34c432db324c","Type":"ContainerStarted","Data":"a4ff3dc46e4fd296cfc877029397226f309df4dbc03e3ae559d07b96fcbe3d41"}
Sep 10 13:39:55.161313 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:55.160881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:39:55.182977 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:39:55.182924    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc" podStartSLOduration=1.715733545 podStartE2EDuration="2.182907231s" totalImagesPullingTime="467.173686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:39:55.181641098 +0000 UTC m=+1300.198473071" watchObservedRunningTime="2026-09-10 13:39:55.182907231 +0000 UTC m=+1300.199739203"
Sep 10 13:40:01.171770 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:01.171744    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:40:07.760406 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:07.760360    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"]
Sep 10 13:40:07.760870 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:07.760697    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz" podUID="92ddfe99-db81-4b6e-8848-409cdd16e19d" containerName="maas-api" containerID="cri-o://5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b" gracePeriod=30
Sep 10 13:40:07.849941 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:07.849905    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"]
Sep 10 13:40:07.850488 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:07.850456    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn" podUID="cbcda5e1-fbf9-4902-941f-defd28ec25cc" containerName="payload-processing" containerID="cri-o://4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a" gracePeriod=30
Sep 10 13:40:07.854264 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:07.854239    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"]
Sep 10 13:40:07.854513 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:07.854453    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42" podUID="38c0584c-d13e-4e12-b0f7-df7cdabdd156" containerName="payload-pre-processing" containerID="cri-o://8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a" gracePeriod=30
Sep 10 13:40:08.126741 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.126713    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:40:08.151036 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.151013    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:40:08.158049 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.158021    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:40:08.213165 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213134    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcda5e1-fbf9-4902-941f-defd28ec25cc-plugins-config-volume\") pod \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\" (UID: \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\") "
Sep 10 13:40:08.213374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213181    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-maas-api-tls\") pod \"92ddfe99-db81-4b6e-8848-409cdd16e19d\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") "
Sep 10 13:40:08.213374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213216    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqkmb\" (UniqueName: \"kubernetes.io/projected/cbcda5e1-fbf9-4902-941f-defd28ec25cc-kube-api-access-gqkmb\") pod \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\" (UID: \"cbcda5e1-fbf9-4902-941f-defd28ec25cc\") "
Sep 10 13:40:08.213374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213261    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-metrics-certs\") pod \"92ddfe99-db81-4b6e-8848-409cdd16e19d\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") "
Sep 10 13:40:08.213374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213294    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c0584c-d13e-4e12-b0f7-df7cdabdd156-plugins-config-volume\") pod \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\" (UID: \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\") "
Sep 10 13:40:08.213374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213356    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ddfe99-db81-4b6e-8848-409cdd16e19d-openshift-service-ca\") pod \"92ddfe99-db81-4b6e-8848-409cdd16e19d\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") "
Sep 10 13:40:08.213658 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213407    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtjg8\" (UniqueName: \"kubernetes.io/projected/38c0584c-d13e-4e12-b0f7-df7cdabdd156-kube-api-access-wtjg8\") pod \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\" (UID: \"38c0584c-d13e-4e12-b0f7-df7cdabdd156\") "
Sep 10 13:40:08.213658 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213439    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f42t\" (UniqueName: \"kubernetes.io/projected/92ddfe99-db81-4b6e-8848-409cdd16e19d-kube-api-access-5f42t\") pod \"92ddfe99-db81-4b6e-8848-409cdd16e19d\" (UID: \"92ddfe99-db81-4b6e-8848-409cdd16e19d\") "
Sep 10 13:40:08.213658 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcda5e1-fbf9-4902-941f-defd28ec25cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cbcda5e1-fbf9-4902-941f-defd28ec25cc" (UID: "cbcda5e1-fbf9-4902-941f-defd28ec25cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:40:08.213860 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213840    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcda5e1-fbf9-4902-941f-defd28ec25cc-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:40:08.213950 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213908    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ddfe99-db81-4b6e-8848-409cdd16e19d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "92ddfe99-db81-4b6e-8848-409cdd16e19d" (UID: "92ddfe99-db81-4b6e-8848-409cdd16e19d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:40:08.214021 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.213962    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c0584c-d13e-4e12-b0f7-df7cdabdd156-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "38c0584c-d13e-4e12-b0f7-df7cdabdd156" (UID: "38c0584c-d13e-4e12-b0f7-df7cdabdd156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:40:08.215768 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.215735    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ddfe99-db81-4b6e-8848-409cdd16e19d-kube-api-access-5f42t" (OuterVolumeSpecName: "kube-api-access-5f42t") pod "92ddfe99-db81-4b6e-8848-409cdd16e19d" (UID: "92ddfe99-db81-4b6e-8848-409cdd16e19d"). InnerVolumeSpecName "kube-api-access-5f42t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:40:08.216175 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.216151    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "92ddfe99-db81-4b6e-8848-409cdd16e19d" (UID: "92ddfe99-db81-4b6e-8848-409cdd16e19d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:40:08.216175 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.216154    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcda5e1-fbf9-4902-941f-defd28ec25cc-kube-api-access-gqkmb" (OuterVolumeSpecName: "kube-api-access-gqkmb") pod "cbcda5e1-fbf9-4902-941f-defd28ec25cc" (UID: "cbcda5e1-fbf9-4902-941f-defd28ec25cc"). InnerVolumeSpecName "kube-api-access-gqkmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:40:08.216332 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.216176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "92ddfe99-db81-4b6e-8848-409cdd16e19d" (UID: "92ddfe99-db81-4b6e-8848-409cdd16e19d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:40:08.216332 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.216241    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c0584c-d13e-4e12-b0f7-df7cdabdd156-kube-api-access-wtjg8" (OuterVolumeSpecName: "kube-api-access-wtjg8") pod "38c0584c-d13e-4e12-b0f7-df7cdabdd156" (UID: "38c0584c-d13e-4e12-b0f7-df7cdabdd156"). InnerVolumeSpecName "kube-api-access-wtjg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:40:08.237879 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.237849    2578 generic.go:356] "Generic (PLEG): container finished" podID="92ddfe99-db81-4b6e-8848-409cdd16e19d" containerID="5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b" exitCode=0
Sep 10 13:40:08.238121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.237897    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"
Sep 10 13:40:08.238121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.237940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz" event={"ID":"92ddfe99-db81-4b6e-8848-409cdd16e19d","Type":"ContainerDied","Data":"5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b"}
Sep 10 13:40:08.238121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.238007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz" event={"ID":"92ddfe99-db81-4b6e-8848-409cdd16e19d","Type":"ContainerDied","Data":"d6af352315e019d57af7f2c8e21e47961a16a4a9598bda511a940bcbaf4521b3"}
Sep 10 13:40:08.238121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.238041    2578 scope.go:117] "RemoveContainer" containerID="5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b"
Sep 10 13:40:08.239339 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.239318    2578 generic.go:356] "Generic (PLEG): container finished" podID="38c0584c-d13e-4e12-b0f7-df7cdabdd156" containerID="8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a" exitCode=0
Sep 10 13:40:08.239449 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.239364    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"
Sep 10 13:40:08.239449 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.239393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42" event={"ID":"38c0584c-d13e-4e12-b0f7-df7cdabdd156","Type":"ContainerDied","Data":"8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a"}
Sep 10 13:40:08.239563 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.239448    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42" event={"ID":"38c0584c-d13e-4e12-b0f7-df7cdabdd156","Type":"ContainerDied","Data":"370cfe9b82f871b932173bdb03f35fe211618f50b05844017bdbd7b3b7b63279"}
Sep 10 13:40:08.240847 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.240824    2578 generic.go:356] "Generic (PLEG): container finished" podID="cbcda5e1-fbf9-4902-941f-defd28ec25cc" containerID="4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a" exitCode=0
Sep 10 13:40:08.240963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.240881    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"
Sep 10 13:40:08.240963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.240915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn" event={"ID":"cbcda5e1-fbf9-4902-941f-defd28ec25cc","Type":"ContainerDied","Data":"4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a"}
Sep 10 13:40:08.241041 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.240966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn" event={"ID":"cbcda5e1-fbf9-4902-941f-defd28ec25cc","Type":"ContainerDied","Data":"41f8aed5c8ae641c8155f68120d9bd58ffad3829c0b5a786555e7917b3575900"}
Sep 10 13:40:08.247545 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.247521    2578 scope.go:117] "RemoveContainer" containerID="5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b"
Sep 10 13:40:08.247877 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:40:08.247855    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b\": container with ID starting with 5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b not found: ID does not exist" containerID="5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b"
Sep 10 13:40:08.247949 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.247903    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b"} err="failed to get container status \"5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b\": rpc error: code = NotFound desc = could not find container \"5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b\": container with ID starting with 5c689cd4b52b94d1d19bebf79ba38a2a59e94bac9c8345e14fc3cfa8ef816d1b not found: ID does not exist"
Sep 10 13:40:08.247949 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.247930    2578 scope.go:117] "RemoveContainer" containerID="8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a"
Sep 10 13:40:08.257671 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.257651    2578 scope.go:117] "RemoveContainer" containerID="8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a"
Sep 10 13:40:08.257946 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:40:08.257928    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a\": container with ID starting with 8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a not found: ID does not exist" containerID="8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a"
Sep 10 13:40:08.257996 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.257952    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a"} err="failed to get container status \"8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a\": rpc error: code = NotFound desc = could not find container \"8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a\": container with ID starting with 8056f8e48ae6f35235e2adf5b12c5c4da7b5149e5ca1895586eca3116d25d24a not found: ID does not exist"
Sep 10 13:40:08.257996 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.257970    2578 scope.go:117] "RemoveContainer" containerID="4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a"
Sep 10 13:40:08.265587 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.265570    2578 scope.go:117] "RemoveContainer" containerID="4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a"
Sep 10 13:40:08.265858 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:40:08.265840    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a\": container with ID starting with 4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a not found: ID does not exist" containerID="4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a"
Sep 10 13:40:08.265913 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.265864    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a"} err="failed to get container status \"4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a\": rpc error: code = NotFound desc = could not find container \"4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a\": container with ID starting with 4a838e75d44193a846a41d10ced89b86a9a49e2e6ee181930c8a476e36346a0a not found: ID does not exist"
Sep 10 13:40:08.274967 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.274893    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"]
Sep 10 13:40:08.279538 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.279512    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-839f01ff-865dc68d65-77tnn"]
Sep 10 13:40:08.294832 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.294800    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"]
Sep 10 13:40:08.298380 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.298354    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-839f01ff-56bdfb7b4b-7xd42"]
Sep 10 13:40:08.313564 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.313541    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"]
Sep 10 13:40:08.314980 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.314960    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ddfe99-db81-4b6e-8848-409cdd16e19d-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:40:08.315038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.314983    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtjg8\" (UniqueName: \"kubernetes.io/projected/38c0584c-d13e-4e12-b0f7-df7cdabdd156-kube-api-access-wtjg8\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:40:08.315038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.314994    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f42t\" (UniqueName: \"kubernetes.io/projected/92ddfe99-db81-4b6e-8848-409cdd16e19d-kube-api-access-5f42t\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:40:08.315038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.315004    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:40:08.315038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.315013    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqkmb\" (UniqueName: \"kubernetes.io/projected/cbcda5e1-fbf9-4902-941f-defd28ec25cc-kube-api-access-gqkmb\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:40:08.315038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.315021    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92ddfe99-db81-4b6e-8848-409cdd16e19d-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:40:08.315038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.315029    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c0584c-d13e-4e12-b0f7-df7cdabdd156-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:40:08.318053 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:08.318032    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-839f01ff-5bc586cd9f-qstcz"]
Sep 10 13:40:09.451433 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:09.451399    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38c0584c-d13e-4e12-b0f7-df7cdabdd156" path="/var/lib/kubelet/pods/38c0584c-d13e-4e12-b0f7-df7cdabdd156/volumes"
Sep 10 13:40:09.451818 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:09.451804    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92ddfe99-db81-4b6e-8848-409cdd16e19d" path="/var/lib/kubelet/pods/92ddfe99-db81-4b6e-8848-409cdd16e19d/volumes"
Sep 10 13:40:09.452113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:09.452101    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbcda5e1-fbf9-4902-941f-defd28ec25cc" path="/var/lib/kubelet/pods/cbcda5e1-fbf9-4902-941f-defd28ec25cc/volumes"
Sep 10 13:40:38.167764 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.167668    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"]
Sep 10 13:40:38.168358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.168090    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ddfe99-db81-4b6e-8848-409cdd16e19d" containerName="maas-api"
Sep 10 13:40:38.168358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.168103    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ddfe99-db81-4b6e-8848-409cdd16e19d" containerName="maas-api"
Sep 10 13:40:38.168358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.168114    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38c0584c-d13e-4e12-b0f7-df7cdabdd156" containerName="payload-pre-processing"
Sep 10 13:40:38.168358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.168120    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38c0584c-d13e-4e12-b0f7-df7cdabdd156" containerName="payload-pre-processing"
Sep 10 13:40:38.168358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.168145    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbcda5e1-fbf9-4902-941f-defd28ec25cc" containerName="payload-processing"
Sep 10 13:40:38.168358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.168154    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbcda5e1-fbf9-4902-941f-defd28ec25cc" containerName="payload-processing"
Sep 10 13:40:38.168358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.168245    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ddfe99-db81-4b6e-8848-409cdd16e19d" containerName="maas-api"
Sep 10 13:40:38.168358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.168256    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbcda5e1-fbf9-4902-941f-defd28ec25cc" containerName="payload-processing"
Sep 10 13:40:38.168358 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.168263    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="38c0584c-d13e-4e12-b0f7-df7cdabdd156" containerName="payload-pre-processing"
Sep 10 13:40:38.171877 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.171853    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.174885 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.174859    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5d75ac-serving-cert\""
Sep 10 13:40:38.190058 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.190032    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"]
Sep 10 13:40:38.279831 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.279790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v29sn\" (UniqueName: \"kubernetes.io/projected/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-kube-api-access-v29sn\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.280015 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.279903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.280015 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.279925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.280094 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.280003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.381087 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.381057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.381087 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.381092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.381475 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.381125    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.381475 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.381155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v29sn\" (UniqueName: \"kubernetes.io/projected/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-kube-api-access-v29sn\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.381872 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.381843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.384220 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.384176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.384322 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.384275    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.395874 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.395845    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v29sn\" (UniqueName: \"kubernetes.io/projected/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-kube-api-access-v29sn\") pod \"maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.483820 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.483787    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:38.817269 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:38.817245    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"]
Sep 10 13:40:38.819998 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:40:38.819955    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7a97fe_b12d_47f5_bc25_075108f2d3a1.slice/crio-bb4290074fab463716f57bc3630383d28492864e0245ac8213274f79c3610aee WatchSource:0}: Error finding container bb4290074fab463716f57bc3630383d28492864e0245ac8213274f79c3610aee: Status 404 returned error can't find the container with id bb4290074fab463716f57bc3630383d28492864e0245ac8213274f79c3610aee
Sep 10 13:40:39.096893 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.096814    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"]
Sep 10 13:40:39.101694 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.101663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:40:39.104597 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.104566    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5d75ac\""
Sep 10 13:40:39.104755 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.104617    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5d75ac-dockercfg-mn8h2\""
Sep 10 13:40:39.114360 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.114336    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"]
Sep 10 13:40:39.187348 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.187313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62hnf\" (UniqueName: \"kubernetes.io/projected/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-kube-api-access-62hnf\") pod \"payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94\" (UID: \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:40:39.187745 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.187382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94\" (UID: \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:40:39.288685 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.288640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62hnf\" (UniqueName: \"kubernetes.io/projected/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-kube-api-access-62hnf\") pod \"payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94\" (UID: \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:40:39.288891 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.288789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94\" (UID: \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:40:39.289486 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.289466    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94\" (UID: \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:40:39.297929 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.297907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62hnf\" (UniqueName: \"kubernetes.io/projected/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-kube-api-access-62hnf\") pod \"payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94\" (UID: \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:40:39.393749 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.393721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx" event={"ID":"7b7a97fe-b12d-47f5-bc25-075108f2d3a1","Type":"ContainerStarted","Data":"bb4290074fab463716f57bc3630383d28492864e0245ac8213274f79c3610aee"}
Sep 10 13:40:39.414270 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.414248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:40:39.693538 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:39.693155    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"]
Sep 10 13:40:39.696001 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:40:39.695968    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913bf8cc_d118_43b6_a8df_9e9bf8e4b163.slice/crio-d4a12f7072153523eb7d4e234312cd259f6fa2b4089e70d93afe56d05cbfffdb WatchSource:0}: Error finding container d4a12f7072153523eb7d4e234312cd259f6fa2b4089e70d93afe56d05cbfffdb: Status 404 returned error can't find the container with id d4a12f7072153523eb7d4e234312cd259f6fa2b4089e70d93afe56d05cbfffdb
Sep 10 13:40:40.399899 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:40.399857    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94" event={"ID":"913bf8cc-d118-43b6-a8df-9e9bf8e4b163","Type":"ContainerStarted","Data":"7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974"}
Sep 10 13:40:40.400376 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:40.399920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94" event={"ID":"913bf8cc-d118-43b6-a8df-9e9bf8e4b163","Type":"ContainerStarted","Data":"d4a12f7072153523eb7d4e234312cd259f6fa2b4089e70d93afe56d05cbfffdb"}
Sep 10 13:40:40.400376 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:40.399952    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:40:40.401713 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:40.401679    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx" event={"ID":"7b7a97fe-b12d-47f5-bc25-075108f2d3a1","Type":"ContainerStarted","Data":"93e3b358e94a63e2f92df1d6c6da402af0c6550affa71865188509767e10a9b4"}
Sep 10 13:40:40.401857 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:40.401740    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:40.448753 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:40.448700    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94" podStartSLOduration=1.448686439 podStartE2EDuration="1.448686439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:40.447551134 +0000 UTC m=+1345.464383107" watchObservedRunningTime="2026-09-10 13:40:40.448686439 +0000 UTC m=+1345.465518410"
Sep 10 13:40:40.512244 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:40.512159    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx" podStartSLOduration=1.946143341 podStartE2EDuration="2.512141662s" totalImagesPullingTime="565.998321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:40:40.511046265 +0000 UTC m=+1345.527878249" watchObservedRunningTime="2026-09-10 13:40:40.512141662 +0000 UTC m=+1345.528973634"
Sep 10 13:40:46.414831 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:46.414798    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:40:51.411093 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:40:51.411054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:41:06.956580 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:06.956542    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"]
Sep 10 13:41:06.960801 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:06.960777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:06.963245 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:06.963189    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-ba174e-serving-cert\""
Sep 10 13:41:06.972151 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:06.972125    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"]
Sep 10 13:41:07.030431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.030397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464bfe84-41c3-48e7-bb59-006f176e79d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.030599 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.030445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.030599 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.030471    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.030599 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.030544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xldbp\" (UniqueName: \"kubernetes.io/projected/464bfe84-41c3-48e7-bb59-006f176e79d6-kube-api-access-xldbp\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.132012 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.131969    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464bfe84-41c3-48e7-bb59-006f176e79d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.132217 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.132034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.132217 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.132068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.132217 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.132098    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xldbp\" (UniqueName: \"kubernetes.io/projected/464bfe84-41c3-48e7-bb59-006f176e79d6-kube-api-access-xldbp\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.132377 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:41:07.132235    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-serving-cert: secret "maas-api-e2e-shared-b-w2-ba174e-serving-cert" not found
Sep 10 13:41:07.132377 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:41:07.132319    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-maas-api-tls podName:464bfe84-41c3-48e7-bb59-006f176e79d6 nodeName:}" failed. No retries permitted until 2026-09-10 13:41:07.632296369 +0000 UTC m=+1372.649128322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-maas-api-tls") pod "maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2" (UID: "464bfe84-41c3-48e7-bb59-006f176e79d6") : secret "maas-api-e2e-shared-b-w2-ba174e-serving-cert" not found
Sep 10 13:41:07.132751 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.132722    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464bfe84-41c3-48e7-bb59-006f176e79d6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.134530 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.134509    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.141430 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.141410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xldbp\" (UniqueName: \"kubernetes.io/projected/464bfe84-41c3-48e7-bb59-006f176e79d6-kube-api-access-xldbp\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.636021 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.635980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.638595 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.638566    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:07.872758 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:07.872727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:08.063284 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.063250    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"]
Sep 10 13:41:08.067011 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.066987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:08.069690 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.069664    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-ba174e-dockercfg-nkx64\""
Sep 10 13:41:08.070072 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.070056    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-ba174e\""
Sep 10 13:41:08.077848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.077826    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"]
Sep 10 13:41:08.140002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.139970    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn28f\" (UniqueName: \"kubernetes.io/projected/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-kube-api-access-nn28f\") pod \"payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8\" (UID: \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:08.140186 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.140021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8\" (UID: \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:08.174862 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.174838    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"]
Sep 10 13:41:08.177170 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:41:08.177140    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464bfe84_41c3_48e7_bb59_006f176e79d6.slice/crio-89e204f1933a098c17a37c1b790189ac79959ca69e1ebb52adf76ada0fe6d5a7 WatchSource:0}: Error finding container 89e204f1933a098c17a37c1b790189ac79959ca69e1ebb52adf76ada0fe6d5a7: Status 404 returned error can't find the container with id 89e204f1933a098c17a37c1b790189ac79959ca69e1ebb52adf76ada0fe6d5a7
Sep 10 13:41:08.240697 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.240660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn28f\" (UniqueName: \"kubernetes.io/projected/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-kube-api-access-nn28f\") pod \"payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8\" (UID: \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:08.240874 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.240713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8\" (UID: \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:08.241424 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.241406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8\" (UID: \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:08.253475 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.253450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn28f\" (UniqueName: \"kubernetes.io/projected/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-kube-api-access-nn28f\") pod \"payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8\" (UID: \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:08.378951 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.378909    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:08.545373 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.545336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2" event={"ID":"464bfe84-41c3-48e7-bb59-006f176e79d6","Type":"ContainerStarted","Data":"89e204f1933a098c17a37c1b790189ac79959ca69e1ebb52adf76ada0fe6d5a7"}
Sep 10 13:41:08.662096 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:08.662067    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"]
Sep 10 13:41:08.662913 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:41:08.662891    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a1aa8c_2db7_4b8e_9d9c_eee4115bbfd2.slice/crio-76492742cdc1cc774db274394732a1e963896f199f22704280a963fe6703cb52 WatchSource:0}: Error finding container 76492742cdc1cc774db274394732a1e963896f199f22704280a963fe6703cb52: Status 404 returned error can't find the container with id 76492742cdc1cc774db274394732a1e963896f199f22704280a963fe6703cb52
Sep 10 13:41:09.551809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:09.551765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2" event={"ID":"464bfe84-41c3-48e7-bb59-006f176e79d6","Type":"ContainerStarted","Data":"c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93"}
Sep 10 13:41:09.552298 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:09.551835    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:09.553459 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:09.553437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8" event={"ID":"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2","Type":"ContainerStarted","Data":"dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5"}
Sep 10 13:41:09.553548 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:09.553474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8" event={"ID":"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2","Type":"ContainerStarted","Data":"76492742cdc1cc774db274394732a1e963896f199f22704280a963fe6703cb52"}
Sep 10 13:41:09.553587 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:09.553574    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:09.573410 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:09.573363    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2" podStartSLOduration=2.9859120409999997 podStartE2EDuration="3.573347345s" totalImagesPullingTime="587.435304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:09.572987942 +0000 UTC m=+1374.589819916" watchObservedRunningTime="2026-09-10 13:41:09.573347345 +0000 UTC m=+1374.590179317"
Sep 10 13:41:09.596769 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:09.596718    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8" podStartSLOduration=1.596700621 podStartE2EDuration="1.596700621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:09.595807745 +0000 UTC m=+1374.612639718" watchObservedRunningTime="2026-09-10 13:41:09.596700621 +0000 UTC m=+1374.613532620"
Sep 10 13:41:15.564108 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:15.564077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:41:20.561372 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:20.561337    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:41:45.353918 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.353883    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"]
Sep 10 13:41:45.357626 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.357604    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.360036 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.360009    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9469c838-serving-cert\""
Sep 10 13:41:45.368308 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.368286    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"]
Sep 10 13:41:45.372302 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.372271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-maas-api-tls\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.372430 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.372311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-metrics-certs\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.372430 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.372399    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clsdb\" (UniqueName: \"kubernetes.io/projected/9324a5bb-b851-4492-b526-f3078a286e89-kube-api-access-clsdb\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.372556 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.372459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324a5bb-b851-4492-b526-f3078a286e89-openshift-service-ca\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.473479 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.473451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-maas-api-tls\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.473479 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.473481    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-metrics-certs\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.473691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.473504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clsdb\" (UniqueName: \"kubernetes.io/projected/9324a5bb-b851-4492-b526-f3078a286e89-kube-api-access-clsdb\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.473691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.473538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324a5bb-b851-4492-b526-f3078a286e89-openshift-service-ca\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.473691 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:41:45.473620    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-serving-cert: secret "maas-api-e2e-mt-9469c838-serving-cert" not found
Sep 10 13:41:45.473796 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:41:45.473705    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-maas-api-tls podName:9324a5bb-b851-4492-b526-f3078a286e89 nodeName:}" failed. No retries permitted until 2026-09-10 13:41:45.973686171 +0000 UTC m=+1410.990518132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-maas-api-tls") pod "maas-api-e2e-mt-9469c838-ff96679dd-l8lgh" (UID: "9324a5bb-b851-4492-b526-f3078a286e89") : secret "maas-api-e2e-mt-9469c838-serving-cert" not found
Sep 10 13:41:45.474113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.474093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324a5bb-b851-4492-b526-f3078a286e89-openshift-service-ca\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.475967 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.475945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-metrics-certs\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.485536 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.485512    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clsdb\" (UniqueName: \"kubernetes.io/projected/9324a5bb-b851-4492-b526-f3078a286e89-kube-api-access-clsdb\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.979504 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.979467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-maas-api-tls\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:45.982016 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:45.981982    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-maas-api-tls\") pod \"maas-api-e2e-mt-9469c838-ff96679dd-l8lgh\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:46.270979 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:46.270896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:46.560173 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:46.560149    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"]
Sep 10 13:41:46.563008 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:41:46.562983    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9324a5bb_b851_4492_b526_f3078a286e89.slice/crio-b639063586b213083a2bbd6bd1d93ebea4ce1a74da7c376ed79c21a3ea4960a2 WatchSource:0}: Error finding container b639063586b213083a2bbd6bd1d93ebea4ce1a74da7c376ed79c21a3ea4960a2: Status 404 returned error can't find the container with id b639063586b213083a2bbd6bd1d93ebea4ce1a74da7c376ed79c21a3ea4960a2
Sep 10 13:41:46.745291 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:46.745251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh" event={"ID":"9324a5bb-b851-4492-b526-f3078a286e89","Type":"ContainerStarted","Data":"b639063586b213083a2bbd6bd1d93ebea4ce1a74da7c376ed79c21a3ea4960a2"}
Sep 10 13:41:47.751520 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:47.751485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh" event={"ID":"9324a5bb-b851-4492-b526-f3078a286e89","Type":"ContainerStarted","Data":"d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20"}
Sep 10 13:41:47.751912 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:47.751570    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:41:47.780393 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:47.780337    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh" podStartSLOduration=2.044962752 podStartE2EDuration="2.780319618s" totalImagesPullingTime="735.356866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:41:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:41:47.778753704 +0000 UTC m=+1412.795585670" watchObservedRunningTime="2026-09-10 13:41:47.780319618 +0000 UTC m=+1412.797151591"
Sep 10 13:41:53.762464 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:41:53.762434    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:42:15.590075 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:15.590002    2578 scope.go:117] "RemoveContainer" containerID="8e6f6abf921ca2b94fd965f7c5cdd7beb511630e646079784b52e27aa466f47f"
Sep 10 13:42:59.100985 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.100295    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"]
Sep 10 13:42:59.101562 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.101237    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh" podUID="9324a5bb-b851-4492-b526-f3078a286e89" containerName="maas-api" containerID="cri-o://d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20" gracePeriod=30
Sep 10 13:42:59.358145 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.358085    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:42:59.464328 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.464301    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clsdb\" (UniqueName: \"kubernetes.io/projected/9324a5bb-b851-4492-b526-f3078a286e89-kube-api-access-clsdb\") pod \"9324a5bb-b851-4492-b526-f3078a286e89\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") "
Sep 10 13:42:59.464510 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.464369    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-maas-api-tls\") pod \"9324a5bb-b851-4492-b526-f3078a286e89\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") "
Sep 10 13:42:59.464510 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.464404    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324a5bb-b851-4492-b526-f3078a286e89-openshift-service-ca\") pod \"9324a5bb-b851-4492-b526-f3078a286e89\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") "
Sep 10 13:42:59.464510 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.464459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-metrics-certs\") pod \"9324a5bb-b851-4492-b526-f3078a286e89\" (UID: \"9324a5bb-b851-4492-b526-f3078a286e89\") "
Sep 10 13:42:59.464772 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.464742    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9324a5bb-b851-4492-b526-f3078a286e89-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9324a5bb-b851-4492-b526-f3078a286e89" (UID: "9324a5bb-b851-4492-b526-f3078a286e89"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:42:59.466356 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.466332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9324a5bb-b851-4492-b526-f3078a286e89" (UID: "9324a5bb-b851-4492-b526-f3078a286e89"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:42:59.466528 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.466511    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9324a5bb-b851-4492-b526-f3078a286e89" (UID: "9324a5bb-b851-4492-b526-f3078a286e89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:42:59.466628 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.466606    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9324a5bb-b851-4492-b526-f3078a286e89-kube-api-access-clsdb" (OuterVolumeSpecName: "kube-api-access-clsdb") pod "9324a5bb-b851-4492-b526-f3078a286e89" (UID: "9324a5bb-b851-4492-b526-f3078a286e89"). InnerVolumeSpecName "kube-api-access-clsdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:42:59.565935 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.565902    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clsdb\" (UniqueName: \"kubernetes.io/projected/9324a5bb-b851-4492-b526-f3078a286e89-kube-api-access-clsdb\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:42:59.565935 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.565928    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:42:59.565935 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.565938    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9324a5bb-b851-4492-b526-f3078a286e89-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:42:59.566243 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:42:59.565947    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9324a5bb-b851-4492-b526-f3078a286e89-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:43:00.124078 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:00.124042    2578 generic.go:356] "Generic (PLEG): container finished" podID="9324a5bb-b851-4492-b526-f3078a286e89" containerID="d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20" exitCode=0
Sep 10 13:43:00.124523 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:00.124087    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"
Sep 10 13:43:00.124523 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:00.124125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh" event={"ID":"9324a5bb-b851-4492-b526-f3078a286e89","Type":"ContainerDied","Data":"d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20"}
Sep 10 13:43:00.124523 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:00.124178    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh" event={"ID":"9324a5bb-b851-4492-b526-f3078a286e89","Type":"ContainerDied","Data":"b639063586b213083a2bbd6bd1d93ebea4ce1a74da7c376ed79c21a3ea4960a2"}
Sep 10 13:43:00.124523 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:00.124221    2578 scope.go:117] "RemoveContainer" containerID="d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20"
Sep 10 13:43:00.134810 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:00.134789    2578 scope.go:117] "RemoveContainer" containerID="d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20"
Sep 10 13:43:00.135092 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:43:00.135076    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20\": container with ID starting with d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20 not found: ID does not exist" containerID="d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20"
Sep 10 13:43:00.135143 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:00.135099    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20"} err="failed to get container status \"d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20\": rpc error: code = NotFound desc = could not find container \"d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20\": container with ID starting with d416bad95467d8b5c68bcf06e5b5418f4d4d933b96b870a447fde4b8cb3d4f20 not found: ID does not exist"
Sep 10 13:43:00.150179 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:00.150153    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"]
Sep 10 13:43:00.155013 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:00.154991    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9469c838-ff96679dd-l8lgh"]
Sep 10 13:43:01.452287 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:43:01.452251    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9324a5bb-b851-4492-b526-f3078a286e89" path="/var/lib/kubelet/pods/9324a5bb-b851-4492-b526-f3078a286e89/volumes"
Sep 10 13:44:52.908331 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:52.908277    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"]
Sep 10 13:44:52.910234 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:52.908879    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9324a5bb-b851-4492-b526-f3078a286e89" containerName="maas-api"
Sep 10 13:44:52.910234 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:52.908895    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9324a5bb-b851-4492-b526-f3078a286e89" containerName="maas-api"
Sep 10 13:44:52.910234 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:52.908961    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9324a5bb-b851-4492-b526-f3078a286e89" containerName="maas-api"
Sep 10 13:44:52.911400 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:52.911382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:52.913689 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:52.913668    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a2d945-serving-cert\""
Sep 10 13:44:52.922577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:52.922549    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"]
Sep 10 13:44:53.072356 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.072318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-metrics-certs\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.072565 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.072388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9fee78-1a67-4227-9de8-cc6edac59d05-openshift-service-ca\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.072565 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.072488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjgxd\" (UniqueName: \"kubernetes.io/projected/fc9fee78-1a67-4227-9de8-cc6edac59d05-kube-api-access-zjgxd\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.072565 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.072538    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-maas-api-tls\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.173829 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.173742    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9fee78-1a67-4227-9de8-cc6edac59d05-openshift-service-ca\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.173829 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.173790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgxd\" (UniqueName: \"kubernetes.io/projected/fc9fee78-1a67-4227-9de8-cc6edac59d05-kube-api-access-zjgxd\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.173829 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.173817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-maas-api-tls\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.174072 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.173875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-metrics-certs\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.174072 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:44:53.173975    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-serving-cert: secret "maas-api-e2e-api-a-a2d945-serving-cert" not found
Sep 10 13:44:53.174072 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:44:53.174046    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-maas-api-tls podName:fc9fee78-1a67-4227-9de8-cc6edac59d05 nodeName:}" failed. No retries permitted until 2026-09-10 13:44:53.674027396 +0000 UTC m=+1598.690859362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-maas-api-tls") pod "maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7" (UID: "fc9fee78-1a67-4227-9de8-cc6edac59d05") : secret "maas-api-e2e-api-a-a2d945-serving-cert" not found
Sep 10 13:44:53.174458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.174437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9fee78-1a67-4227-9de8-cc6edac59d05-openshift-service-ca\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.176397 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.176377    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-metrics-certs\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.183801 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.183768    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgxd\" (UniqueName: \"kubernetes.io/projected/fc9fee78-1a67-4227-9de8-cc6edac59d05-kube-api-access-zjgxd\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.678512 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.678478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-maas-api-tls\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.681013 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.680984    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-maas-api-tls\") pod \"maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:53.824984 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:53.824940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:54.098772 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:54.098745    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"]
Sep 10 13:44:54.101179 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:44:54.101148    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9fee78_1a67_4227_9de8_cc6edac59d05.slice/crio-25ce6531d983031ca2045fec623031fdac74d914ab02bf7bc117f26eb38ade5e WatchSource:0}: Error finding container 25ce6531d983031ca2045fec623031fdac74d914ab02bf7bc117f26eb38ade5e: Status 404 returned error can't find the container with id 25ce6531d983031ca2045fec623031fdac74d914ab02bf7bc117f26eb38ade5e
Sep 10 13:44:54.102767 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:54.102745    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:44:54.722635 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:54.722600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7" event={"ID":"fc9fee78-1a67-4227-9de8-cc6edac59d05","Type":"ContainerStarted","Data":"8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543"}
Sep 10 13:44:54.722859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:54.722657    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7" event={"ID":"fc9fee78-1a67-4227-9de8-cc6edac59d05","Type":"ContainerStarted","Data":"25ce6531d983031ca2045fec623031fdac74d914ab02bf7bc117f26eb38ade5e"}
Sep 10 13:44:54.722859 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:54.722705    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:44:54.742522 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:44:54.742465    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7" podStartSLOduration=2.244910728 podStartE2EDuration="2.742447719s" totalImagesPullingTime="497.536991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:44:54.741591655 +0000 UTC m=+1599.758423626" watchObservedRunningTime="2026-09-10 13:44:54.742447719 +0000 UTC m=+1599.759279744"
Sep 10 13:45:00.732839 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:00.732807    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:45:22.277801 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.277765    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"]
Sep 10 13:45:22.282205 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.282171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.284641 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.284616    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-02a403-serving-cert\""
Sep 10 13:45:22.293598 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.293572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"]
Sep 10 13:45:22.343876 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.343838    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a977aff-505a-4018-8594-12d6b009c383-openshift-service-ca\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.344075 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.343887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-maas-api-tls\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.344075 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.343956    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgt4g\" (UniqueName: \"kubernetes.io/projected/1a977aff-505a-4018-8594-12d6b009c383-kube-api-access-sgt4g\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.344075 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.344026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-metrics-certs\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.445366 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.445335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a977aff-505a-4018-8594-12d6b009c383-openshift-service-ca\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.445568 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.445378    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-maas-api-tls\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.445568 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.445396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgt4g\" (UniqueName: \"kubernetes.io/projected/1a977aff-505a-4018-8594-12d6b009c383-kube-api-access-sgt4g\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.445568 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.445430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-metrics-certs\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.446061 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.446035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a977aff-505a-4018-8594-12d6b009c383-openshift-service-ca\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.447898 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.447878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-metrics-certs\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.447995 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.447976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-maas-api-tls\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.457591 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.457571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgt4g\" (UniqueName: \"kubernetes.io/projected/1a977aff-505a-4018-8594-12d6b009c383-kube-api-access-sgt4g\") pod \"maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.595562 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.595478    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:22.873662 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:22.873639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"]
Sep 10 13:45:22.875216 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:45:22.875164    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a977aff_505a_4018_8594_12d6b009c383.slice/crio-908884a44dae9cfd6d2568c3ba294962259b6a327ae8fdb3aef10c7dfe2ac748 WatchSource:0}: Error finding container 908884a44dae9cfd6d2568c3ba294962259b6a327ae8fdb3aef10c7dfe2ac748: Status 404 returned error can't find the container with id 908884a44dae9cfd6d2568c3ba294962259b6a327ae8fdb3aef10c7dfe2ac748
Sep 10 13:45:23.878723 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:23.878684    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz" event={"ID":"1a977aff-505a-4018-8594-12d6b009c383","Type":"ContainerStarted","Data":"324d5eca781f304a99e5fc35ce3152a841d117f28d81c050698e05b7f33f98ff"}
Sep 10 13:45:23.879114 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:23.878743    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz" event={"ID":"1a977aff-505a-4018-8594-12d6b009c383","Type":"ContainerStarted","Data":"908884a44dae9cfd6d2568c3ba294962259b6a327ae8fdb3aef10c7dfe2ac748"}
Sep 10 13:45:23.879114 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:23.878895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:45:23.901027 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:23.900969    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz" podStartSLOduration=1.432833958 podStartE2EDuration="1.900949942s" totalImagesPullingTime="468.115984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:45:23.900243854 +0000 UTC m=+1628.917075824" watchObservedRunningTime="2026-09-10 13:45:23.900949942 +0000 UTC m=+1628.917781919"
Sep 10 13:45:29.889746 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:45:29.889719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:46:04.782647 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:04.782606    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"]
Sep 10 13:46:04.783246 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:04.782864    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7" podUID="fc9fee78-1a67-4227-9de8-cc6edac59d05" containerName="maas-api" containerID="cri-o://8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543" gracePeriod=30
Sep 10 13:46:05.069381 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.069354    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:46:05.090745 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.090704    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc9fee78-1a67-4227-9de8-cc6edac59d05" containerID="8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543" exitCode=0
Sep 10 13:46:05.090959 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.090829    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7" event={"ID":"fc9fee78-1a67-4227-9de8-cc6edac59d05","Type":"ContainerDied","Data":"8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543"}
Sep 10 13:46:05.090959 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.090846    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"
Sep 10 13:46:05.090959 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.090920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7" event={"ID":"fc9fee78-1a67-4227-9de8-cc6edac59d05","Type":"ContainerDied","Data":"25ce6531d983031ca2045fec623031fdac74d914ab02bf7bc117f26eb38ade5e"}
Sep 10 13:46:05.090959 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.090954    2578 scope.go:117] "RemoveContainer" containerID="8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543"
Sep 10 13:46:05.101954 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.101905    2578 scope.go:117] "RemoveContainer" containerID="8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543"
Sep 10 13:46:05.102440 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:46:05.102406    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543\": container with ID starting with 8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543 not found: ID does not exist" containerID="8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543"
Sep 10 13:46:05.102541 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.102454    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543"} err="failed to get container status \"8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543\": rpc error: code = NotFound desc = could not find container \"8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543\": container with ID starting with 8d25d3d2de81e7ae1ec306f0f4bacd9245a431bf670d2f1c327120acf1ec7543 not found: ID does not exist"
Sep 10 13:46:05.136965 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.136930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-metrics-certs\") pod \"fc9fee78-1a67-4227-9de8-cc6edac59d05\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") "
Sep 10 13:46:05.137167 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.136980    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-maas-api-tls\") pod \"fc9fee78-1a67-4227-9de8-cc6edac59d05\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") "
Sep 10 13:46:05.137167 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.137012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9fee78-1a67-4227-9de8-cc6edac59d05-openshift-service-ca\") pod \"fc9fee78-1a67-4227-9de8-cc6edac59d05\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") "
Sep 10 13:46:05.137167 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.137078    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjgxd\" (UniqueName: \"kubernetes.io/projected/fc9fee78-1a67-4227-9de8-cc6edac59d05-kube-api-access-zjgxd\") pod \"fc9fee78-1a67-4227-9de8-cc6edac59d05\" (UID: \"fc9fee78-1a67-4227-9de8-cc6edac59d05\") "
Sep 10 13:46:05.137432 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.137405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9fee78-1a67-4227-9de8-cc6edac59d05-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc9fee78-1a67-4227-9de8-cc6edac59d05" (UID: "fc9fee78-1a67-4227-9de8-cc6edac59d05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:46:05.139228 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.139173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc9fee78-1a67-4227-9de8-cc6edac59d05" (UID: "fc9fee78-1a67-4227-9de8-cc6edac59d05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:05.139332 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.139229    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fc9fee78-1a67-4227-9de8-cc6edac59d05" (UID: "fc9fee78-1a67-4227-9de8-cc6edac59d05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:05.139332 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.139253    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9fee78-1a67-4227-9de8-cc6edac59d05-kube-api-access-zjgxd" (OuterVolumeSpecName: "kube-api-access-zjgxd") pod "fc9fee78-1a67-4227-9de8-cc6edac59d05" (UID: "fc9fee78-1a67-4227-9de8-cc6edac59d05"). InnerVolumeSpecName "kube-api-access-zjgxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:05.238735 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.238701    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.238735 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.238729    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9fee78-1a67-4227-9de8-cc6edac59d05-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.238735 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.238741    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9fee78-1a67-4227-9de8-cc6edac59d05-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.238979 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.238751    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjgxd\" (UniqueName: \"kubernetes.io/projected/fc9fee78-1a67-4227-9de8-cc6edac59d05-kube-api-access-zjgxd\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:05.415723 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.415685    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"]
Sep 10 13:46:05.420775 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.420734    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a2d945-648c5d45f8-snsb7"]
Sep 10 13:46:05.452525 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:05.452490    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc9fee78-1a67-4227-9de8-cc6edac59d05" path="/var/lib/kubelet/pods/fc9fee78-1a67-4227-9de8-cc6edac59d05/volumes"
Sep 10 13:46:28.067539 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.067506    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"]
Sep 10 13:46:28.068365 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.068311    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz" podUID="1a977aff-505a-4018-8594-12d6b009c383" containerName="maas-api" containerID="cri-o://324d5eca781f304a99e5fc35ce3152a841d117f28d81c050698e05b7f33f98ff" gracePeriod=30
Sep 10 13:46:28.223897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.223860    2578 generic.go:356] "Generic (PLEG): container finished" podID="1a977aff-505a-4018-8594-12d6b009c383" containerID="324d5eca781f304a99e5fc35ce3152a841d117f28d81c050698e05b7f33f98ff" exitCode=0
Sep 10 13:46:28.224069 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.223927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz" event={"ID":"1a977aff-505a-4018-8594-12d6b009c383","Type":"ContainerDied","Data":"324d5eca781f304a99e5fc35ce3152a841d117f28d81c050698e05b7f33f98ff"}
Sep 10 13:46:28.329277 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.329189    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:46:28.454183 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.454148    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a977aff-505a-4018-8594-12d6b009c383-openshift-service-ca\") pod \"1a977aff-505a-4018-8594-12d6b009c383\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") "
Sep 10 13:46:28.454374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.454249    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-maas-api-tls\") pod \"1a977aff-505a-4018-8594-12d6b009c383\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") "
Sep 10 13:46:28.454374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.454294    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-metrics-certs\") pod \"1a977aff-505a-4018-8594-12d6b009c383\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") "
Sep 10 13:46:28.454374 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.454340    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgt4g\" (UniqueName: \"kubernetes.io/projected/1a977aff-505a-4018-8594-12d6b009c383-kube-api-access-sgt4g\") pod \"1a977aff-505a-4018-8594-12d6b009c383\" (UID: \"1a977aff-505a-4018-8594-12d6b009c383\") "
Sep 10 13:46:28.454559 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.454519    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a977aff-505a-4018-8594-12d6b009c383-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a977aff-505a-4018-8594-12d6b009c383" (UID: "1a977aff-505a-4018-8594-12d6b009c383"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:46:28.454654 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.454637    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a977aff-505a-4018-8594-12d6b009c383-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:28.456546 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.456515    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a977aff-505a-4018-8594-12d6b009c383" (UID: "1a977aff-505a-4018-8594-12d6b009c383"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:28.456679 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.456567    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a977aff-505a-4018-8594-12d6b009c383" (UID: "1a977aff-505a-4018-8594-12d6b009c383"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:28.456679 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.456658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a977aff-505a-4018-8594-12d6b009c383-kube-api-access-sgt4g" (OuterVolumeSpecName: "kube-api-access-sgt4g") pod "1a977aff-505a-4018-8594-12d6b009c383" (UID: "1a977aff-505a-4018-8594-12d6b009c383"). InnerVolumeSpecName "kube-api-access-sgt4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:28.556097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.556059    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:28.556097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.556089    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a977aff-505a-4018-8594-12d6b009c383-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:28.556097 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:28.556101    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgt4g\" (UniqueName: \"kubernetes.io/projected/1a977aff-505a-4018-8594-12d6b009c383-kube-api-access-sgt4g\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:29.231072 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:29.231038    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"
Sep 10 13:46:29.231559 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:29.231040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz" event={"ID":"1a977aff-505a-4018-8594-12d6b009c383","Type":"ContainerDied","Data":"908884a44dae9cfd6d2568c3ba294962259b6a327ae8fdb3aef10c7dfe2ac748"}
Sep 10 13:46:29.231559 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:29.231187    2578 scope.go:117] "RemoveContainer" containerID="324d5eca781f304a99e5fc35ce3152a841d117f28d81c050698e05b7f33f98ff"
Sep 10 13:46:29.256089 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:29.256060    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"]
Sep 10 13:46:29.260133 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:29.260108    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-02a403-58dcc96d8d-hxjvz"]
Sep 10 13:46:29.453169 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:29.453135    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a977aff-505a-4018-8594-12d6b009c383" path="/var/lib/kubelet/pods/1a977aff-505a-4018-8594-12d6b009c383/volumes"
Sep 10 13:46:46.173511 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.173429    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"]
Sep 10 13:46:46.173966 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.173698    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx" podUID="7b7a97fe-b12d-47f5-bc25-075108f2d3a1" containerName="maas-api" containerID="cri-o://93e3b358e94a63e2f92df1d6c6da402af0c6550affa71865188509767e10a9b4" gracePeriod=30
Sep 10 13:46:46.214737 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.214701    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"]
Sep 10 13:46:46.215161 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.214968    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94" podUID="913bf8cc-d118-43b6-a8df-9e9bf8e4b163" containerName="payload-processing" containerID="cri-o://7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974" gracePeriod=30
Sep 10 13:46:46.324754 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.324722    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b7a97fe-b12d-47f5-bc25-075108f2d3a1" containerID="93e3b358e94a63e2f92df1d6c6da402af0c6550affa71865188509767e10a9b4" exitCode=0
Sep 10 13:46:46.324886 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.324792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx" event={"ID":"7b7a97fe-b12d-47f5-bc25-075108f2d3a1","Type":"ContainerDied","Data":"93e3b358e94a63e2f92df1d6c6da402af0c6550affa71865188509767e10a9b4"}
Sep 10 13:46:46.478086 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.478059    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:46:46.500549 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.500480    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-openshift-service-ca\") pod \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") "
Sep 10 13:46:46.500708 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.500647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v29sn\" (UniqueName: \"kubernetes.io/projected/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-kube-api-access-v29sn\") pod \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") "
Sep 10 13:46:46.500708 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.500694    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-metrics-certs\") pod \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") "
Sep 10 13:46:46.500837 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.500724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-maas-api-tls\") pod \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\" (UID: \"7b7a97fe-b12d-47f5-bc25-075108f2d3a1\") "
Sep 10 13:46:46.500837 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.500821    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b7a97fe-b12d-47f5-bc25-075108f2d3a1" (UID: "7b7a97fe-b12d-47f5-bc25-075108f2d3a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:46:46.501324 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.501072    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.503682 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.503654    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b7a97fe-b12d-47f5-bc25-075108f2d3a1" (UID: "7b7a97fe-b12d-47f5-bc25-075108f2d3a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:46.504585 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.504559    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b7a97fe-b12d-47f5-bc25-075108f2d3a1" (UID: "7b7a97fe-b12d-47f5-bc25-075108f2d3a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:46:46.505692 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.505665    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-kube-api-access-v29sn" (OuterVolumeSpecName: "kube-api-access-v29sn") pod "7b7a97fe-b12d-47f5-bc25-075108f2d3a1" (UID: "7b7a97fe-b12d-47f5-bc25-075108f2d3a1"). InnerVolumeSpecName "kube-api-access-v29sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:46.507421 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.507399    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:46:46.601591 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.601557    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-plugins-config-volume\") pod \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\" (UID: \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\") "
Sep 10 13:46:46.601772 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.601619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62hnf\" (UniqueName: \"kubernetes.io/projected/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-kube-api-access-62hnf\") pod \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\" (UID: \"913bf8cc-d118-43b6-a8df-9e9bf8e4b163\") "
Sep 10 13:46:46.601816 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.601779    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v29sn\" (UniqueName: \"kubernetes.io/projected/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-kube-api-access-v29sn\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.601816 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.601791    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.601816 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.601800    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b7a97fe-b12d-47f5-bc25-075108f2d3a1-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.602031 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.602001    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "913bf8cc-d118-43b6-a8df-9e9bf8e4b163" (UID: "913bf8cc-d118-43b6-a8df-9e9bf8e4b163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:46:46.603809 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.603780    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-kube-api-access-62hnf" (OuterVolumeSpecName: "kube-api-access-62hnf") pod "913bf8cc-d118-43b6-a8df-9e9bf8e4b163" (UID: "913bf8cc-d118-43b6-a8df-9e9bf8e4b163"). InnerVolumeSpecName "kube-api-access-62hnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:46:46.702259 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.702164    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62hnf\" (UniqueName: \"kubernetes.io/projected/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-kube-api-access-62hnf\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:46.702259 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:46.702205    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/913bf8cc-d118-43b6-a8df-9e9bf8e4b163-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:46:47.331433 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.331399    2578 generic.go:356] "Generic (PLEG): container finished" podID="913bf8cc-d118-43b6-a8df-9e9bf8e4b163" containerID="7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974" exitCode=0
Sep 10 13:46:47.331897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.331444    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"
Sep 10 13:46:47.331897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.331491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94" event={"ID":"913bf8cc-d118-43b6-a8df-9e9bf8e4b163","Type":"ContainerDied","Data":"7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974"}
Sep 10 13:46:47.331897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.331544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94" event={"ID":"913bf8cc-d118-43b6-a8df-9e9bf8e4b163","Type":"ContainerDied","Data":"d4a12f7072153523eb7d4e234312cd259f6fa2b4089e70d93afe56d05cbfffdb"}
Sep 10 13:46:47.331897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.331576    2578 scope.go:117] "RemoveContainer" containerID="7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974"
Sep 10 13:46:47.333120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.332966    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx" event={"ID":"7b7a97fe-b12d-47f5-bc25-075108f2d3a1","Type":"ContainerDied","Data":"bb4290074fab463716f57bc3630383d28492864e0245ac8213274f79c3610aee"}
Sep 10 13:46:47.333120 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.332997    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"
Sep 10 13:46:47.340607 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.340585    2578 scope.go:117] "RemoveContainer" containerID="7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974"
Sep 10 13:46:47.341060 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:46:47.340928    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974\": container with ID starting with 7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974 not found: ID does not exist" containerID="7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974"
Sep 10 13:46:47.341060 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.340963    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974"} err="failed to get container status \"7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974\": rpc error: code = NotFound desc = could not find container \"7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974\": container with ID starting with 7448cb1f48afff7f3b6558cf9aa011f92aa165a43e0dff01f43bbde4bfb5f974 not found: ID does not exist"
Sep 10 13:46:47.341060 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.340984    2578 scope.go:117] "RemoveContainer" containerID="93e3b358e94a63e2f92df1d6c6da402af0c6550affa71865188509767e10a9b4"
Sep 10 13:46:47.357514 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.357486    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"]
Sep 10 13:46:47.360854 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.360828    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx"]
Sep 10 13:46:47.377133 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.373436    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"]
Sep 10 13:46:47.379599 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.379569    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5d75ac-7d95d69999-scw94"]
Sep 10 13:46:47.451687 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.451605    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b7a97fe-b12d-47f5-bc25-075108f2d3a1" path="/var/lib/kubelet/pods/7b7a97fe-b12d-47f5-bc25-075108f2d3a1/volumes"
Sep 10 13:46:47.451971 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:47.451957    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="913bf8cc-d118-43b6-a8df-9e9bf8e4b163" path="/var/lib/kubelet/pods/913bf8cc-d118-43b6-a8df-9e9bf8e4b163/volumes"
Sep 10 13:46:49.409449 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:46:49.409403    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5d75ac-9d8f9c6c7-9vhxx" podUID="7b7a97fe-b12d-47f5-bc25-075108f2d3a1" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.56:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 10 13:47:03.852465 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.852431    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"]
Sep 10 13:47:03.853028 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853007    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b7a97fe-b12d-47f5-bc25-075108f2d3a1" containerName="maas-api"
Sep 10 13:47:03.853112 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853031    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b7a97fe-b12d-47f5-bc25-075108f2d3a1" containerName="maas-api"
Sep 10 13:47:03.853112 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853046    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9fee78-1a67-4227-9de8-cc6edac59d05" containerName="maas-api"
Sep 10 13:47:03.853112 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853055    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9fee78-1a67-4227-9de8-cc6edac59d05" containerName="maas-api"
Sep 10 13:47:03.853112 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853095    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a977aff-505a-4018-8594-12d6b009c383" containerName="maas-api"
Sep 10 13:47:03.853112 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853104    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a977aff-505a-4018-8594-12d6b009c383" containerName="maas-api"
Sep 10 13:47:03.853395 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853115    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="913bf8cc-d118-43b6-a8df-9e9bf8e4b163" containerName="payload-processing"
Sep 10 13:47:03.853395 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853123    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="913bf8cc-d118-43b6-a8df-9e9bf8e4b163" containerName="payload-processing"
Sep 10 13:47:03.853395 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853230    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="913bf8cc-d118-43b6-a8df-9e9bf8e4b163" containerName="payload-processing"
Sep 10 13:47:03.853395 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853250    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9fee78-1a67-4227-9de8-cc6edac59d05" containerName="maas-api"
Sep 10 13:47:03.853395 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853259    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a977aff-505a-4018-8594-12d6b009c383" containerName="maas-api"
Sep 10 13:47:03.853395 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.853267    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b7a97fe-b12d-47f5-bc25-075108f2d3a1" containerName="maas-api"
Sep 10 13:47:03.857929 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.857902    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:03.860866 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.860842    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-41abaf-serving-cert\""
Sep 10 13:47:03.867609 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.867083    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"]
Sep 10 13:47:03.958685 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.958647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-metrics-certs\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:03.958866 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.958704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:03.958866 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.958730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z647g\" (UniqueName: \"kubernetes.io/projected/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-kube-api-access-z647g\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:03.958866 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:03.958752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.059866 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.059823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-metrics-certs\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.060081 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.059904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.060081 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.059927    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z647g\" (UniqueName: \"kubernetes.io/projected/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-kube-api-access-z647g\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.060081 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.059950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.060081 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:47:04.060056    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-serving-cert: secret "maas-api-e2e-ipp-a-41abaf-serving-cert" not found
Sep 10 13:47:04.060337 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:47:04.060127    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-maas-api-tls podName:4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762 nodeName:}" failed. No retries permitted until 2026-09-10 13:47:04.560103662 +0000 UTC m=+1729.576935626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-maas-api-tls") pod "maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4" (UID: "4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762") : secret "maas-api-e2e-ipp-a-41abaf-serving-cert" not found
Sep 10 13:47:04.060688 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.060667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.062305 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.062282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-metrics-certs\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.074034 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.074002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z647g\" (UniqueName: \"kubernetes.io/projected/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-kube-api-access-z647g\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.564737 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.564693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.567317 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.567294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.656141 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.656098    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"]
Sep 10 13:47:04.660140 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.660114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:04.662829 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.662803    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-41abaf\""
Sep 10 13:47:04.662829 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.662830    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-41abaf-dockercfg-r5hlz\""
Sep 10 13:47:04.671053 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.671025    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"]
Sep 10 13:47:04.766941 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.766903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hzt\" (UniqueName: \"kubernetes.io/projected/81269b8b-8532-494f-be96-afe0e17ed5ff-kube-api-access-m6hzt\") pod \"payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh\" (UID: \"81269b8b-8532-494f-be96-afe0e17ed5ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:04.767121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.766975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81269b8b-8532-494f-be96-afe0e17ed5ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh\" (UID: \"81269b8b-8532-494f-be96-afe0e17ed5ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:04.771828 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.771799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:04.848976 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.848942    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"]
Sep 10 13:47:04.849844 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.849188    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2" podUID="464bfe84-41c3-48e7-bb59-006f176e79d6" containerName="maas-api" containerID="cri-o://c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93" gracePeriod=30
Sep 10 13:47:04.868587 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.868509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81269b8b-8532-494f-be96-afe0e17ed5ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh\" (UID: \"81269b8b-8532-494f-be96-afe0e17ed5ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:04.868994 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.868648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hzt\" (UniqueName: \"kubernetes.io/projected/81269b8b-8532-494f-be96-afe0e17ed5ff-kube-api-access-m6hzt\") pod \"payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh\" (UID: \"81269b8b-8532-494f-be96-afe0e17ed5ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:04.870379 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.870286    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81269b8b-8532-494f-be96-afe0e17ed5ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh\" (UID: \"81269b8b-8532-494f-be96-afe0e17ed5ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:04.880566 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.880532    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hzt\" (UniqueName: \"kubernetes.io/projected/81269b8b-8532-494f-be96-afe0e17ed5ff-kube-api-access-m6hzt\") pod \"payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh\" (UID: \"81269b8b-8532-494f-be96-afe0e17ed5ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:04.914030 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.913993    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"]
Sep 10 13:47:04.914832 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.914524    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8" podUID="c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2" containerName="payload-processing" containerID="cri-o://dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5" gracePeriod=30
Sep 10 13:47:04.973269 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:04.973237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:05.058955 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.058842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"]
Sep 10 13:47:05.216322 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.211477    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:47:05.251264 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.251236    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:47:05.264295 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.263927    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"]
Sep 10 13:47:05.266456 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:47:05.266424    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81269b8b_8532_494f_be96_afe0e17ed5ff.slice/crio-31adfc122fe7466fd4ca1ffcebf5ce6850d91294941f4cbe2d59b728bbefa4cf WatchSource:0}: Error finding container 31adfc122fe7466fd4ca1ffcebf5ce6850d91294941f4cbe2d59b728bbefa4cf: Status 404 returned error can't find the container with id 31adfc122fe7466fd4ca1ffcebf5ce6850d91294941f4cbe2d59b728bbefa4cf
Sep 10 13:47:05.272564 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.272538    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xldbp\" (UniqueName: \"kubernetes.io/projected/464bfe84-41c3-48e7-bb59-006f176e79d6-kube-api-access-xldbp\") pod \"464bfe84-41c3-48e7-bb59-006f176e79d6\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") "
Sep 10 13:47:05.272691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.272594    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-metrics-certs\") pod \"464bfe84-41c3-48e7-bb59-006f176e79d6\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") "
Sep 10 13:47:05.272691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.272646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-maas-api-tls\") pod \"464bfe84-41c3-48e7-bb59-006f176e79d6\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") "
Sep 10 13:47:05.272807 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.272690    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464bfe84-41c3-48e7-bb59-006f176e79d6-openshift-service-ca\") pod \"464bfe84-41c3-48e7-bb59-006f176e79d6\" (UID: \"464bfe84-41c3-48e7-bb59-006f176e79d6\") "
Sep 10 13:47:05.273163 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.273130    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464bfe84-41c3-48e7-bb59-006f176e79d6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "464bfe84-41c3-48e7-bb59-006f176e79d6" (UID: "464bfe84-41c3-48e7-bb59-006f176e79d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:47:05.275382 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.275355    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "464bfe84-41c3-48e7-bb59-006f176e79d6" (UID: "464bfe84-41c3-48e7-bb59-006f176e79d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:47:05.275474 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.275392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464bfe84-41c3-48e7-bb59-006f176e79d6-kube-api-access-xldbp" (OuterVolumeSpecName: "kube-api-access-xldbp") pod "464bfe84-41c3-48e7-bb59-006f176e79d6" (UID: "464bfe84-41c3-48e7-bb59-006f176e79d6"). InnerVolumeSpecName "kube-api-access-xldbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:47:05.275587 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.275567    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "464bfe84-41c3-48e7-bb59-006f176e79d6" (UID: "464bfe84-41c3-48e7-bb59-006f176e79d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:47:05.373536 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.373455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-plugins-config-volume\") pod \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\" (UID: \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\") "
Sep 10 13:47:05.373715 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.373579    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nn28f\" (UniqueName: \"kubernetes.io/projected/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-kube-api-access-nn28f\") pod \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\" (UID: \"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2\") "
Sep 10 13:47:05.373782 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.373765    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:47:05.373850 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.373786    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464bfe84-41c3-48e7-bb59-006f176e79d6-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:47:05.373850 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.373800    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464bfe84-41c3-48e7-bb59-006f176e79d6-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:47:05.373850 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.373813    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xldbp\" (UniqueName: \"kubernetes.io/projected/464bfe84-41c3-48e7-bb59-006f176e79d6-kube-api-access-xldbp\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:47:05.374001 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.373950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2" (UID: "c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:47:05.375672 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.375636    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-kube-api-access-nn28f" (OuterVolumeSpecName: "kube-api-access-nn28f") pod "c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2" (UID: "c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2"). InnerVolumeSpecName "kube-api-access-nn28f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:47:05.427264 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.427227    2578 generic.go:356] "Generic (PLEG): container finished" podID="464bfe84-41c3-48e7-bb59-006f176e79d6" containerID="c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93" exitCode=0
Sep 10 13:47:05.427468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.427316    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2" event={"ID":"464bfe84-41c3-48e7-bb59-006f176e79d6","Type":"ContainerDied","Data":"c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93"}
Sep 10 13:47:05.427468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.427324    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"
Sep 10 13:47:05.427468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.427375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2" event={"ID":"464bfe84-41c3-48e7-bb59-006f176e79d6","Type":"ContainerDied","Data":"89e204f1933a098c17a37c1b790189ac79959ca69e1ebb52adf76ada0fe6d5a7"}
Sep 10 13:47:05.427468 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.427410    2578 scope.go:117] "RemoveContainer" containerID="c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93"
Sep 10 13:47:05.428857 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.428831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4" event={"ID":"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762","Type":"ContainerStarted","Data":"373778ab5ff83c27fcc9597979ba36ad071987bd93ec89d98eeb1c6c1970fdb1"}
Sep 10 13:47:05.430486 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.430460    2578 generic.go:356] "Generic (PLEG): container finished" podID="c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2" containerID="dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5" exitCode=0
Sep 10 13:47:05.430656 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.430504    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"
Sep 10 13:47:05.430656 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.430542    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8" event={"ID":"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2","Type":"ContainerDied","Data":"dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5"}
Sep 10 13:47:05.430656 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.430584    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8" event={"ID":"c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2","Type":"ContainerDied","Data":"76492742cdc1cc774db274394732a1e963896f199f22704280a963fe6703cb52"}
Sep 10 13:47:05.432420 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.432337    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh" event={"ID":"81269b8b-8532-494f-be96-afe0e17ed5ff","Type":"ContainerStarted","Data":"0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f"}
Sep 10 13:47:05.432420 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.432386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh" event={"ID":"81269b8b-8532-494f-be96-afe0e17ed5ff","Type":"ContainerStarted","Data":"31adfc122fe7466fd4ca1ffcebf5ce6850d91294941f4cbe2d59b728bbefa4cf"}
Sep 10 13:47:05.432620 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.432468    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:05.441687 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.441548    2578 scope.go:117] "RemoveContainer" containerID="c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93"
Sep 10 13:47:05.442139 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:47:05.442101    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93\": container with ID starting with c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93 not found: ID does not exist" containerID="c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93"
Sep 10 13:47:05.442276 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.442140    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93"} err="failed to get container status \"c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93\": rpc error: code = NotFound desc = could not find container \"c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93\": container with ID starting with c4ec11529fd1de491e8a787644be830b98d980b0ea9abfe973d4c43b44e20d93 not found: ID does not exist"
Sep 10 13:47:05.442276 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.442163    2578 scope.go:117] "RemoveContainer" containerID="dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5"
Sep 10 13:47:05.457536 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.457485    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh" podStartSLOduration=1.457468652 podStartE2EDuration="1.457468652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:05.456605393 +0000 UTC m=+1730.473437385" watchObservedRunningTime="2026-09-10 13:47:05.457468652 +0000 UTC m=+1730.474300625"
Sep 10 13:47:05.459576 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.459558    2578 scope.go:117] "RemoveContainer" containerID="dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5"
Sep 10 13:47:05.459893 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:47:05.459875    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5\": container with ID starting with dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5 not found: ID does not exist" containerID="dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5"
Sep 10 13:47:05.459946 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.459900    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5"} err="failed to get container status \"dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5\": rpc error: code = NotFound desc = could not find container \"dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5\": container with ID starting with dd1a34aff0173383d3e54e5a1a2a0c12f4664bc02d41d7e9477a5cb5331e20b5 not found: ID does not exist"
Sep 10 13:47:05.474669 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.474639    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:47:05.474669 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.474668    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn28f\" (UniqueName: \"kubernetes.io/projected/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2-kube-api-access-nn28f\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:47:05.481582 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.481554    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"]
Sep 10 13:47:05.489660 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.489634    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-ba174e-766cc76499-thsx8"]
Sep 10 13:47:05.505360 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.505328    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"]
Sep 10 13:47:05.510434 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:05.510402    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-ba174e-65c8bf9c79-9cpn2"]
Sep 10 13:47:06.440631 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:06.440537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4" event={"ID":"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762","Type":"ContainerStarted","Data":"ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0"}
Sep 10 13:47:06.441043 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:06.440734    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:06.459450 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:06.459396    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4" podStartSLOduration=3.063029781 podStartE2EDuration="3.459378624s" totalImagesPullingTime="396.348843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:06.458394667 +0000 UTC m=+1731.475226638" watchObservedRunningTime="2026-09-10 13:47:06.459378624 +0000 UTC m=+1731.476210596"
Sep 10 13:47:07.451974 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:07.451933    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464bfe84-41c3-48e7-bb59-006f176e79d6" path="/var/lib/kubelet/pods/464bfe84-41c3-48e7-bb59-006f176e79d6/volumes"
Sep 10 13:47:07.452526 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:07.452501    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2" path="/var/lib/kubelet/pods/c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2/volumes"
Sep 10 13:47:12.452105 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:12.452075    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:47:16.443190 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:16.443156    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:47:32.972115 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.972082    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"]
Sep 10 13:47:32.972959 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.972935    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464bfe84-41c3-48e7-bb59-006f176e79d6" containerName="maas-api"
Sep 10 13:47:32.973056 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.972961    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="464bfe84-41c3-48e7-bb59-006f176e79d6" containerName="maas-api"
Sep 10 13:47:32.973056 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.972995    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2" containerName="payload-processing"
Sep 10 13:47:32.973056 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.973005    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2" containerName="payload-processing"
Sep 10 13:47:32.973250 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.973109    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a1aa8c-2db7-4b8e-9d9c-eee4115bbfd2" containerName="payload-processing"
Sep 10 13:47:32.973250 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.973122    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="464bfe84-41c3-48e7-bb59-006f176e79d6" containerName="maas-api"
Sep 10 13:47:32.976261 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.976241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:32.980336 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.980315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-645758-serving-cert\""
Sep 10 13:47:32.992437 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:32.992414    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"]
Sep 10 13:47:33.026485 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.026451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.026485 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.026488    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.026683 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.026511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-metrics-certs\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.026683 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.026607    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5prf8\" (UniqueName: \"kubernetes.io/projected/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-kube-api-access-5prf8\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.127170 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.127133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5prf8\" (UniqueName: \"kubernetes.io/projected/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-kube-api-access-5prf8\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.127396 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.127186    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.127396 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.127241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.127396 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.127274    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-metrics-certs\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.127396 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:47:33.127379    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-serving-cert: secret "maas-api-e2e-ipp-b-645758-serving-cert" not found
Sep 10 13:47:33.127656 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:47:33.127458    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-maas-api-tls podName:bfdc0310-c7da-421d-a3a9-da0c0c7f594c nodeName:}" failed. No retries permitted until 2026-09-10 13:47:33.627440053 +0000 UTC m=+1758.644272003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-maas-api-tls") pod "maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4" (UID: "bfdc0310-c7da-421d-a3a9-da0c0c7f594c") : secret "maas-api-e2e-ipp-b-645758-serving-cert" not found
Sep 10 13:47:33.127912 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.127887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.129855 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.129833    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-metrics-certs\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.162679 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.162646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5prf8\" (UniqueName: \"kubernetes.io/projected/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-kube-api-access-5prf8\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.631292 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.631247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.633587 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.633567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.743934 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.743900    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"]
Sep 10 13:47:33.747945 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.747923    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:47:33.752127 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.752102    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-645758\""
Sep 10 13:47:33.766064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.766034    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"]
Sep 10 13:47:33.833027 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.832992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-645758-864958b789-lbv94\" (UID: \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:47:33.833251 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.833064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-kube-api-access-c2bcg\") pod \"payload-processing-e2e-ipp-b-645758-864958b789-lbv94\" (UID: \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:47:33.886896 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.886813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:33.934556 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.934516    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-kube-api-access-c2bcg\") pod \"payload-processing-e2e-ipp-b-645758-864958b789-lbv94\" (UID: \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:47:33.935025 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.934991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-645758-864958b789-lbv94\" (UID: \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:47:33.935977 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.935919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-645758-864958b789-lbv94\" (UID: \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:47:33.946837 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:33.946806    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-kube-api-access-c2bcg\") pod \"payload-processing-e2e-ipp-b-645758-864958b789-lbv94\" (UID: \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:47:34.058437 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:34.058404    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:47:34.169073 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:34.169043    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"]
Sep 10 13:47:34.170492 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:47:34.170468    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdc0310_c7da_421d_a3a9_da0c0c7f594c.slice/crio-31a098b074405009ea971b46e316e7bb57ec42ed7150b176ebe7ded1b9095389 WatchSource:0}: Error finding container 31a098b074405009ea971b46e316e7bb57ec42ed7150b176ebe7ded1b9095389: Status 404 returned error can't find the container with id 31a098b074405009ea971b46e316e7bb57ec42ed7150b176ebe7ded1b9095389
Sep 10 13:47:34.345346 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:34.345299    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"]
Sep 10 13:47:34.348221 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:47:34.348181    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e25f62_8c4c_407a_b06a_09f6f63e0f35.slice/crio-36f7451fbf4f0a817c1513bfdab43e2dc02cf94c66382a67d50a18841a38e72b WatchSource:0}: Error finding container 36f7451fbf4f0a817c1513bfdab43e2dc02cf94c66382a67d50a18841a38e72b: Status 404 returned error can't find the container with id 36f7451fbf4f0a817c1513bfdab43e2dc02cf94c66382a67d50a18841a38e72b
Sep 10 13:47:34.590614 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:34.590574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4" event={"ID":"bfdc0310-c7da-421d-a3a9-da0c0c7f594c","Type":"ContainerStarted","Data":"31a098b074405009ea971b46e316e7bb57ec42ed7150b176ebe7ded1b9095389"}
Sep 10 13:47:34.592281 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:34.592252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94" event={"ID":"e9e25f62-8c4c-407a-b06a-09f6f63e0f35","Type":"ContainerStarted","Data":"4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6"}
Sep 10 13:47:34.592416 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:34.592299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94" event={"ID":"e9e25f62-8c4c-407a-b06a-09f6f63e0f35","Type":"ContainerStarted","Data":"36f7451fbf4f0a817c1513bfdab43e2dc02cf94c66382a67d50a18841a38e72b"}
Sep 10 13:47:34.592416 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:34.592346    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:47:34.617815 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:34.617758    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94" podStartSLOduration=1.617739577 podStartE2EDuration="1.617739577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:34.615144585 +0000 UTC m=+1759.631976557" watchObservedRunningTime="2026-09-10 13:47:34.617739577 +0000 UTC m=+1759.634571590"
Sep 10 13:47:35.598624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:35.598586    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4" event={"ID":"bfdc0310-c7da-421d-a3a9-da0c0c7f594c","Type":"ContainerStarted","Data":"ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b"}
Sep 10 13:47:35.599076 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:35.598681    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:35.621476 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:35.621416    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4" podStartSLOduration=3.088996796 podStartE2EDuration="3.621398874s" totalImagesPullingTime="532.402078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:47:35.620493209 +0000 UTC m=+1760.637325181" watchObservedRunningTime="2026-09-10 13:47:35.621398874 +0000 UTC m=+1760.638230845"
Sep 10 13:47:41.611044 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:41.611011    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:47:45.601333 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:47:45.601290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:48:01.966576 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:01.966545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cs9hb_2114350c-4310-43c2-9fc9-4ef47e717a06/payload-processing/0.log"
Sep 10 13:48:02.087904 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:02.087868    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-645758-864958b789-lbv94_e9e25f62-8c4c-407a-b06a-09f6f63e0f35/payload-processing/0.log"
Sep 10 13:48:27.636355 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:27.636312    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cs9hb_2114350c-4310-43c2-9fc9-4ef47e717a06/payload-processing/0.log"
Sep 10 13:48:27.769479 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:27.769448    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-645758-864958b789-lbv94_e9e25f62-8c4c-407a-b06a-09f6f63e0f35/payload-processing/0.log"
Sep 10 13:48:32.608357 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.608315    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"]
Sep 10 13:48:32.612236 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.612210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.614846 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.614817    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-08e484-serving-cert\""
Sep 10 13:48:32.627853 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.627824    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"]
Sep 10 13:48:32.662012 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.661971    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.662244 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.662030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sf2w\" (UniqueName: \"kubernetes.io/projected/d7106e4a-e027-4e84-923d-0c554292f85c-kube-api-access-6sf2w\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.662244 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.662076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.662244 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.662107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7106e4a-e027-4e84-923d-0c554292f85c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.763281 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.763243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.763494 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.763303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sf2w\" (UniqueName: \"kubernetes.io/projected/d7106e4a-e027-4e84-923d-0c554292f85c-kube-api-access-6sf2w\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.763494 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.763336    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.763494 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.763364    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7106e4a-e027-4e84-923d-0c554292f85c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.763494 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:48:32.763388    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-serving-cert: secret "maas-api-e2e-ipp-cleanup-08e484-serving-cert" not found
Sep 10 13:48:32.763494 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:48:32.763449    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-maas-api-tls podName:d7106e4a-e027-4e84-923d-0c554292f85c nodeName:}" failed. No retries permitted until 2026-09-10 13:48:33.263432572 +0000 UTC m=+1818.280264538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz" (UID: "d7106e4a-e027-4e84-923d-0c554292f85c") : secret "maas-api-e2e-ipp-cleanup-08e484-serving-cert" not found
Sep 10 13:48:32.764103 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.764078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7106e4a-e027-4e84-923d-0c554292f85c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.765785 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.765766    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:32.773546 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:32.773513    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sf2w\" (UniqueName: \"kubernetes.io/projected/d7106e4a-e027-4e84-923d-0c554292f85c-kube-api-access-6sf2w\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:33.267880 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.267839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:33.270369 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.270335    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:33.453559 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.453524    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"]
Sep 10 13:48:33.457083 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.457064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:48:33.459893 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.459874    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-08e484\""
Sep 10 13:48:33.460003 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.459892    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-08e484-dockercfg-wntm8\""
Sep 10 13:48:33.467285 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.467258    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"]
Sep 10 13:48:33.523125 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.523029    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:33.570559 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.570517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gq8t\" (UniqueName: \"kubernetes.io/projected/2582fe27-9967-4282-be6e-b4f70af2f9fe-kube-api-access-7gq8t\") pod \"payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z\" (UID: \"2582fe27-9967-4282-be6e-b4f70af2f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:48:33.570751 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.570606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2582fe27-9967-4282-be6e-b4f70af2f9fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z\" (UID: \"2582fe27-9967-4282-be6e-b4f70af2f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:48:33.671556 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.671515    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq8t\" (UniqueName: \"kubernetes.io/projected/2582fe27-9967-4282-be6e-b4f70af2f9fe-kube-api-access-7gq8t\") pod \"payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z\" (UID: \"2582fe27-9967-4282-be6e-b4f70af2f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:48:33.671556 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.671569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2582fe27-9967-4282-be6e-b4f70af2f9fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z\" (UID: \"2582fe27-9967-4282-be6e-b4f70af2f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:48:33.672255 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.672238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2582fe27-9967-4282-be6e-b4f70af2f9fe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z\" (UID: \"2582fe27-9967-4282-be6e-b4f70af2f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:48:33.681038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.681013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq8t\" (UniqueName: \"kubernetes.io/projected/2582fe27-9967-4282-be6e-b4f70af2f9fe-kube-api-access-7gq8t\") pod \"payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z\" (UID: \"2582fe27-9967-4282-be6e-b4f70af2f9fe\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:48:33.770294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.770254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:48:33.800100 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.800074    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"]
Sep 10 13:48:33.803579 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:48:33.803542    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7106e4a_e027_4e84_923d_0c554292f85c.slice/crio-77ee67f4f23eef32f390a7a63ecbb23ec5b7d94b6a22c6ff8e4e9cfb36db6d4e WatchSource:0}: Error finding container 77ee67f4f23eef32f390a7a63ecbb23ec5b7d94b6a22c6ff8e4e9cfb36db6d4e: Status 404 returned error can't find the container with id 77ee67f4f23eef32f390a7a63ecbb23ec5b7d94b6a22c6ff8e4e9cfb36db6d4e
Sep 10 13:48:33.896326 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:33.896292    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz" event={"ID":"d7106e4a-e027-4e84-923d-0c554292f85c","Type":"ContainerStarted","Data":"77ee67f4f23eef32f390a7a63ecbb23ec5b7d94b6a22c6ff8e4e9cfb36db6d4e"}
Sep 10 13:48:34.054696 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:34.054671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"]
Sep 10 13:48:34.056612 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:48:34.056570    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2582fe27_9967_4282_be6e_b4f70af2f9fe.slice/crio-f6ed3c4efd7d38f2554eee132dd9a5fbc0dd32de244c2a49fa34c05bc6db680c WatchSource:0}: Error finding container f6ed3c4efd7d38f2554eee132dd9a5fbc0dd32de244c2a49fa34c05bc6db680c: Status 404 returned error can't find the container with id f6ed3c4efd7d38f2554eee132dd9a5fbc0dd32de244c2a49fa34c05bc6db680c
Sep 10 13:48:34.905323 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:34.905272    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z" event={"ID":"2582fe27-9967-4282-be6e-b4f70af2f9fe","Type":"ContainerStarted","Data":"b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e"}
Sep 10 13:48:34.905801 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:34.905377    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z" event={"ID":"2582fe27-9967-4282-be6e-b4f70af2f9fe","Type":"ContainerStarted","Data":"f6ed3c4efd7d38f2554eee132dd9a5fbc0dd32de244c2a49fa34c05bc6db680c"}
Sep 10 13:48:34.905801 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:34.905411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:48:34.907631 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:34.907593    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz" event={"ID":"d7106e4a-e027-4e84-923d-0c554292f85c","Type":"ContainerStarted","Data":"dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494"}
Sep 10 13:48:34.907781 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:34.907661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:34.924479 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:34.924416    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z" podStartSLOduration=1.924390987 podStartE2EDuration="1.924390987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:34.922527579 +0000 UTC m=+1819.939359550" watchObservedRunningTime="2026-09-10 13:48:34.924390987 +0000 UTC m=+1819.941222957"
Sep 10 13:48:34.941721 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:34.941670    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz" podStartSLOduration=2.535486998 podStartE2EDuration="2.941637268s" totalImagesPullingTime="406.15027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:48:34.940414257 +0000 UTC m=+1819.957246229" watchObservedRunningTime="2026-09-10 13:48:34.941637268 +0000 UTC m=+1819.958469239"
Sep 10 13:48:40.919085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:40.919054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:48:45.916076 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:48:45.916042    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:49:03.180362 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.180322    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"]
Sep 10 13:49:03.180835 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.180576    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz" podUID="d7106e4a-e027-4e84-923d-0c554292f85c" containerName="maas-api" containerID="cri-o://dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494" gracePeriod=30
Sep 10 13:49:03.239911 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.239874    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"]
Sep 10 13:49:03.240164 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.240135    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z" podUID="2582fe27-9967-4282-be6e-b4f70af2f9fe" containerName="payload-processing" containerID="cri-o://b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e" gracePeriod=30
Sep 10 13:49:03.503974 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.503943    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:49:03.534959 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.534928    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:49:03.652945 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.652907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-maas-api-tls\") pod \"d7106e4a-e027-4e84-923d-0c554292f85c\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") "
Sep 10 13:49:03.653139 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.652963    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2582fe27-9967-4282-be6e-b4f70af2f9fe-plugins-config-volume\") pod \"2582fe27-9967-4282-be6e-b4f70af2f9fe\" (UID: \"2582fe27-9967-4282-be6e-b4f70af2f9fe\") "
Sep 10 13:49:03.653139 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.653082    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gq8t\" (UniqueName: \"kubernetes.io/projected/2582fe27-9967-4282-be6e-b4f70af2f9fe-kube-api-access-7gq8t\") pod \"2582fe27-9967-4282-be6e-b4f70af2f9fe\" (UID: \"2582fe27-9967-4282-be6e-b4f70af2f9fe\") "
Sep 10 13:49:03.653139 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.653131    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-metrics-certs\") pod \"d7106e4a-e027-4e84-923d-0c554292f85c\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") "
Sep 10 13:49:03.653343 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.653165    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sf2w\" (UniqueName: \"kubernetes.io/projected/d7106e4a-e027-4e84-923d-0c554292f85c-kube-api-access-6sf2w\") pod \"d7106e4a-e027-4e84-923d-0c554292f85c\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") "
Sep 10 13:49:03.653343 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.653182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7106e4a-e027-4e84-923d-0c554292f85c-openshift-service-ca\") pod \"d7106e4a-e027-4e84-923d-0c554292f85c\" (UID: \"d7106e4a-e027-4e84-923d-0c554292f85c\") "
Sep 10 13:49:03.653458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.653391    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2582fe27-9967-4282-be6e-b4f70af2f9fe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2582fe27-9967-4282-be6e-b4f70af2f9fe" (UID: "2582fe27-9967-4282-be6e-b4f70af2f9fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:49:03.653558 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.653529    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2582fe27-9967-4282-be6e-b4f70af2f9fe-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:03.653679 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.653650    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7106e4a-e027-4e84-923d-0c554292f85c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d7106e4a-e027-4e84-923d-0c554292f85c" (UID: "d7106e4a-e027-4e84-923d-0c554292f85c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:49:03.655347 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.655288    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d7106e4a-e027-4e84-923d-0c554292f85c" (UID: "d7106e4a-e027-4e84-923d-0c554292f85c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:03.655347 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.655331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2582fe27-9967-4282-be6e-b4f70af2f9fe-kube-api-access-7gq8t" (OuterVolumeSpecName: "kube-api-access-7gq8t") pod "2582fe27-9967-4282-be6e-b4f70af2f9fe" (UID: "2582fe27-9967-4282-be6e-b4f70af2f9fe"). InnerVolumeSpecName "kube-api-access-7gq8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:03.655587 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.655367    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d7106e4a-e027-4e84-923d-0c554292f85c" (UID: "d7106e4a-e027-4e84-923d-0c554292f85c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:03.655587 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.655405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7106e4a-e027-4e84-923d-0c554292f85c-kube-api-access-6sf2w" (OuterVolumeSpecName: "kube-api-access-6sf2w") pod "d7106e4a-e027-4e84-923d-0c554292f85c" (UID: "d7106e4a-e027-4e84-923d-0c554292f85c"). InnerVolumeSpecName "kube-api-access-6sf2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:03.754448 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.754403    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:03.754448 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.754441    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sf2w\" (UniqueName: \"kubernetes.io/projected/d7106e4a-e027-4e84-923d-0c554292f85c-kube-api-access-6sf2w\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:03.754448 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.754452    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7106e4a-e027-4e84-923d-0c554292f85c-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:03.754717 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.754461    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7106e4a-e027-4e84-923d-0c554292f85c-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:03.754717 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:03.754470    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gq8t\" (UniqueName: \"kubernetes.io/projected/2582fe27-9967-4282-be6e-b4f70af2f9fe-kube-api-access-7gq8t\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:04.069848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.069762    2578 generic.go:356] "Generic (PLEG): container finished" podID="2582fe27-9967-4282-be6e-b4f70af2f9fe" containerID="b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e" exitCode=0
Sep 10 13:49:04.069848 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.069828    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"
Sep 10 13:49:04.070075 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.069846    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z" event={"ID":"2582fe27-9967-4282-be6e-b4f70af2f9fe","Type":"ContainerDied","Data":"b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e"}
Sep 10 13:49:04.070075 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.069927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z" event={"ID":"2582fe27-9967-4282-be6e-b4f70af2f9fe","Type":"ContainerDied","Data":"f6ed3c4efd7d38f2554eee132dd9a5fbc0dd32de244c2a49fa34c05bc6db680c"}
Sep 10 13:49:04.070075 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.069952    2578 scope.go:117] "RemoveContainer" containerID="b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e"
Sep 10 13:49:04.071476 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.071447    2578 generic.go:356] "Generic (PLEG): container finished" podID="d7106e4a-e027-4e84-923d-0c554292f85c" containerID="dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494" exitCode=0
Sep 10 13:49:04.071586 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.071522    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"
Sep 10 13:49:04.071586 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.071531    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz" event={"ID":"d7106e4a-e027-4e84-923d-0c554292f85c","Type":"ContainerDied","Data":"dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494"}
Sep 10 13:49:04.071586 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.071574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz" event={"ID":"d7106e4a-e027-4e84-923d-0c554292f85c","Type":"ContainerDied","Data":"77ee67f4f23eef32f390a7a63ecbb23ec5b7d94b6a22c6ff8e4e9cfb36db6d4e"}
Sep 10 13:49:04.080609 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.080591    2578 scope.go:117] "RemoveContainer" containerID="b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e"
Sep 10 13:49:04.080952 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:49:04.080931    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e\": container with ID starting with b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e not found: ID does not exist" containerID="b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e"
Sep 10 13:49:04.081038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.080965    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e"} err="failed to get container status \"b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e\": rpc error: code = NotFound desc = could not find container \"b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e\": container with ID starting with b59a1d9a1fee5d4a6673bde1cf820b2b5af813f574534327829432a45194e68e not found: ID does not exist"
Sep 10 13:49:04.081038 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.080991    2578 scope.go:117] "RemoveContainer" containerID="dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494"
Sep 10 13:49:04.089609 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.089588    2578 scope.go:117] "RemoveContainer" containerID="dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494"
Sep 10 13:49:04.089890 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:49:04.089868    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494\": container with ID starting with dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494 not found: ID does not exist" containerID="dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494"
Sep 10 13:49:04.089961 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.089896    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494"} err="failed to get container status \"dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494\": rpc error: code = NotFound desc = could not find container \"dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494\": container with ID starting with dcac6221e4f2668e564ad26aa4a3d79dc6e96037ed54d60ca5db4fcaf2a2b494 not found: ID does not exist"
Sep 10 13:49:04.094650 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.094623    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"]
Sep 10 13:49:04.098949 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.098927    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-08e484-555588dbc7-km46z"]
Sep 10 13:49:04.110680 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.110658    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"]
Sep 10 13:49:04.116333 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:04.116311    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-08e484-57f55f84d4-lm2vz"]
Sep 10 13:49:05.452387 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:05.452352    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2582fe27-9967-4282-be6e-b4f70af2f9fe" path="/var/lib/kubelet/pods/2582fe27-9967-4282-be6e-b4f70af2f9fe/volumes"
Sep 10 13:49:05.452770 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:05.452755    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7106e4a-e027-4e84-923d-0c554292f85c" path="/var/lib/kubelet/pods/d7106e4a-e027-4e84-923d-0c554292f85c/volumes"
Sep 10 13:49:30.295507 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.295471    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"]
Sep 10 13:49:30.295982 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.295780    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4" podUID="4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" containerName="maas-api" containerID="cri-o://ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0" gracePeriod=30
Sep 10 13:49:30.346323 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:49:30.346288    2578 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a29dea3_3b4b_44ca_b2b6_44cc9d7e5762.slice/crio-ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a29dea3_3b4b_44ca_b2b6_44cc9d7e5762.slice/crio-conmon-ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 13:49:30.373482 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.373452    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"]
Sep 10 13:49:30.373712 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.373683    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh" podUID="81269b8b-8532-494f-be96-afe0e17ed5ff" containerName="payload-pre-processing" containerID="cri-o://0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f" gracePeriod=30
Sep 10 13:49:30.569021 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.568997    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:49:30.647033 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.647007    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:49:30.711756 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.711724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6hzt\" (UniqueName: \"kubernetes.io/projected/81269b8b-8532-494f-be96-afe0e17ed5ff-kube-api-access-m6hzt\") pod \"81269b8b-8532-494f-be96-afe0e17ed5ff\" (UID: \"81269b8b-8532-494f-be96-afe0e17ed5ff\") "
Sep 10 13:49:30.711912 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.711773    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-metrics-certs\") pod \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") "
Sep 10 13:49:30.711912 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.711829    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-openshift-service-ca\") pod \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") "
Sep 10 13:49:30.711912 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.711867    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-maas-api-tls\") pod \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") "
Sep 10 13:49:30.711912 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.711891    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81269b8b-8532-494f-be96-afe0e17ed5ff-plugins-config-volume\") pod \"81269b8b-8532-494f-be96-afe0e17ed5ff\" (UID: \"81269b8b-8532-494f-be96-afe0e17ed5ff\") "
Sep 10 13:49:30.712137 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.711946    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z647g\" (UniqueName: \"kubernetes.io/projected/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-kube-api-access-z647g\") pod \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\" (UID: \"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762\") "
Sep 10 13:49:30.712211 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.712149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" (UID: "4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:49:30.712415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.712366    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81269b8b-8532-494f-be96-afe0e17ed5ff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81269b8b-8532-494f-be96-afe0e17ed5ff" (UID: "81269b8b-8532-494f-be96-afe0e17ed5ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:49:30.714020 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.713988    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" (UID: "4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:30.714020 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.714014    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" (UID: "4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:49:30.714167 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.714096    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81269b8b-8532-494f-be96-afe0e17ed5ff-kube-api-access-m6hzt" (OuterVolumeSpecName: "kube-api-access-m6hzt") pod "81269b8b-8532-494f-be96-afe0e17ed5ff" (UID: "81269b8b-8532-494f-be96-afe0e17ed5ff"). InnerVolumeSpecName "kube-api-access-m6hzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:30.714401 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.714380    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-kube-api-access-z647g" (OuterVolumeSpecName: "kube-api-access-z647g") pod "4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" (UID: "4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762"). InnerVolumeSpecName "kube-api-access-z647g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:49:30.813068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.812993    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hzt\" (UniqueName: \"kubernetes.io/projected/81269b8b-8532-494f-be96-afe0e17ed5ff-kube-api-access-m6hzt\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:30.813068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.813019    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:30.813068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.813029    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:30.813068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.813038    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:30.813068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.813047    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81269b8b-8532-494f-be96-afe0e17ed5ff-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:30.813068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:30.813056    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z647g\" (UniqueName: \"kubernetes.io/projected/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762-kube-api-access-z647g\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:49:31.212001 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.211919    2578 generic.go:356] "Generic (PLEG): container finished" podID="4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" containerID="ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0" exitCode=0
Sep 10 13:49:31.212001 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.211966    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"
Sep 10 13:49:31.212257 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.212006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4" event={"ID":"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762","Type":"ContainerDied","Data":"ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0"}
Sep 10 13:49:31.212257 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.212063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4" event={"ID":"4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762","Type":"ContainerDied","Data":"373778ab5ff83c27fcc9597979ba36ad071987bd93ec89d98eeb1c6c1970fdb1"}
Sep 10 13:49:31.212257 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.212091    2578 scope.go:117] "RemoveContainer" containerID="ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0"
Sep 10 13:49:31.213546 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.213522    2578 generic.go:356] "Generic (PLEG): container finished" podID="81269b8b-8532-494f-be96-afe0e17ed5ff" containerID="0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f" exitCode=0
Sep 10 13:49:31.213668 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.213562    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"
Sep 10 13:49:31.213668 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.213585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh" event={"ID":"81269b8b-8532-494f-be96-afe0e17ed5ff","Type":"ContainerDied","Data":"0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f"}
Sep 10 13:49:31.213668 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.213624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh" event={"ID":"81269b8b-8532-494f-be96-afe0e17ed5ff","Type":"ContainerDied","Data":"31adfc122fe7466fd4ca1ffcebf5ce6850d91294941f4cbe2d59b728bbefa4cf"}
Sep 10 13:49:31.221343 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.221184    2578 scope.go:117] "RemoveContainer" containerID="ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0"
Sep 10 13:49:31.221634 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:49:31.221613    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0\": container with ID starting with ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0 not found: ID does not exist" containerID="ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0"
Sep 10 13:49:31.221725 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.221644    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0"} err="failed to get container status \"ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0\": rpc error: code = NotFound desc = could not find container \"ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0\": container with ID starting with ae29f0bdf7954a312ff14c5f79aba871e233c29b12ea2baf65d0c19554a85eb0 not found: ID does not exist"
Sep 10 13:49:31.221725 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.221665    2578 scope.go:117] "RemoveContainer" containerID="0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f"
Sep 10 13:49:31.229818 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.229798    2578 scope.go:117] "RemoveContainer" containerID="0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f"
Sep 10 13:49:31.230098 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:49:31.230079    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f\": container with ID starting with 0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f not found: ID does not exist" containerID="0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f"
Sep 10 13:49:31.230147 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.230107    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f"} err="failed to get container status \"0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f\": rpc error: code = NotFound desc = could not find container \"0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f\": container with ID starting with 0d92c8d5f66badce10c797b4af211503d5f841b417b402042fb04a2c089a204f not found: ID does not exist"
Sep 10 13:49:31.237593 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.237570    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"]
Sep 10 13:49:31.242296 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.242274    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41abaf-6cdd688d45-xc6m4"]
Sep 10 13:49:31.253273 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.253250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"]
Sep 10 13:49:31.258000 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.257978    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-41abaf-bd49cffd6-gfbxh"]
Sep 10 13:49:31.452588 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.452554    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" path="/var/lib/kubelet/pods/4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762/volumes"
Sep 10 13:49:31.452947 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:49:31.452896    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81269b8b-8532-494f-be96-afe0e17ed5ff" path="/var/lib/kubelet/pods/81269b8b-8532-494f-be96-afe0e17ed5ff/volumes"
Sep 10 13:50:30.495237 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.495175    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"]
Sep 10 13:50:30.495843 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.495490    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4" podUID="bfdc0310-c7da-421d-a3a9-da0c0c7f594c" containerName="maas-api" containerID="cri-o://ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b" gracePeriod=30
Sep 10 13:50:30.559530 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.559497    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"]
Sep 10 13:50:30.559817 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.559788    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94" podUID="e9e25f62-8c4c-407a-b06a-09f6f63e0f35" containerName="payload-processing" containerID="cri-o://4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6" gracePeriod=30
Sep 10 13:50:30.839733 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.839705    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:50:30.842436 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.842404    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:50:30.953361 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.953319    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-plugins-config-volume\") pod \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\" (UID: \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\") "
Sep 10 13:50:30.953361 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.953363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-kube-api-access-c2bcg\") pod \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\" (UID: \"e9e25f62-8c4c-407a-b06a-09f6f63e0f35\") "
Sep 10 13:50:30.953624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.953414    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-metrics-certs\") pod \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") "
Sep 10 13:50:30.953624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.953433    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5prf8\" (UniqueName: \"kubernetes.io/projected/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-kube-api-access-5prf8\") pod \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") "
Sep 10 13:50:30.953624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.953499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-maas-api-tls\") pod \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") "
Sep 10 13:50:30.953624 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.953539    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-openshift-service-ca\") pod \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\" (UID: \"bfdc0310-c7da-421d-a3a9-da0c0c7f594c\") "
Sep 10 13:50:30.953839 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.953816    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9e25f62-8c4c-407a-b06a-09f6f63e0f35" (UID: "e9e25f62-8c4c-407a-b06a-09f6f63e0f35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:50:30.954068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.954032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bfdc0310-c7da-421d-a3a9-da0c0c7f594c" (UID: "bfdc0310-c7da-421d-a3a9-da0c0c7f594c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:50:30.955747 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.955724    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-kube-api-access-c2bcg" (OuterVolumeSpecName: "kube-api-access-c2bcg") pod "e9e25f62-8c4c-407a-b06a-09f6f63e0f35" (UID: "e9e25f62-8c4c-407a-b06a-09f6f63e0f35"). InnerVolumeSpecName "kube-api-access-c2bcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:50:30.955835 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.955737    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bfdc0310-c7da-421d-a3a9-da0c0c7f594c" (UID: "bfdc0310-c7da-421d-a3a9-da0c0c7f594c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:30.955992 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.955976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bfdc0310-c7da-421d-a3a9-da0c0c7f594c" (UID: "bfdc0310-c7da-421d-a3a9-da0c0c7f594c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:30.956039 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:30.955976    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-kube-api-access-5prf8" (OuterVolumeSpecName: "kube-api-access-5prf8") pod "bfdc0310-c7da-421d-a3a9-da0c0c7f594c" (UID: "bfdc0310-c7da-421d-a3a9-da0c0c7f594c"). InnerVolumeSpecName "kube-api-access-5prf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:50:31.054437 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.054329    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:31.054437 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.054375    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:31.054437 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.054385    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-plugins-config-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:31.054437 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.054395    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2bcg\" (UniqueName: \"kubernetes.io/projected/e9e25f62-8c4c-407a-b06a-09f6f63e0f35-kube-api-access-c2bcg\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:31.054437 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.054403    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:31.054437 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.054411    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5prf8\" (UniqueName: \"kubernetes.io/projected/bfdc0310-c7da-421d-a3a9-da0c0c7f594c-kube-api-access-5prf8\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:31.529204 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.529154    2578 generic.go:356] "Generic (PLEG): container finished" podID="bfdc0310-c7da-421d-a3a9-da0c0c7f594c" containerID="ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b" exitCode=0
Sep 10 13:50:31.529654 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.529227    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"
Sep 10 13:50:31.529654 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.529251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4" event={"ID":"bfdc0310-c7da-421d-a3a9-da0c0c7f594c","Type":"ContainerDied","Data":"ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b"}
Sep 10 13:50:31.529654 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.529312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4" event={"ID":"bfdc0310-c7da-421d-a3a9-da0c0c7f594c","Type":"ContainerDied","Data":"31a098b074405009ea971b46e316e7bb57ec42ed7150b176ebe7ded1b9095389"}
Sep 10 13:50:31.529654 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.529338    2578 scope.go:117] "RemoveContainer" containerID="ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b"
Sep 10 13:50:31.530723 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.530643    2578 generic.go:356] "Generic (PLEG): container finished" podID="e9e25f62-8c4c-407a-b06a-09f6f63e0f35" containerID="4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6" exitCode=0
Sep 10 13:50:31.530723 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.530691    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"
Sep 10 13:50:31.530823 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.530693    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94" event={"ID":"e9e25f62-8c4c-407a-b06a-09f6f63e0f35","Type":"ContainerDied","Data":"4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6"}
Sep 10 13:50:31.530865 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.530826    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94" event={"ID":"e9e25f62-8c4c-407a-b06a-09f6f63e0f35","Type":"ContainerDied","Data":"36f7451fbf4f0a817c1513bfdab43e2dc02cf94c66382a67d50a18841a38e72b"}
Sep 10 13:50:31.538967 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.538952    2578 scope.go:117] "RemoveContainer" containerID="ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b"
Sep 10 13:50:31.539215 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:50:31.539178    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b\": container with ID starting with ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b not found: ID does not exist" containerID="ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b"
Sep 10 13:50:31.539275 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.539217    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b"} err="failed to get container status \"ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b\": rpc error: code = NotFound desc = could not find container \"ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b\": container with ID starting with ce75f62f2040e4a5f665b5c80e16b8fe322569c057899f3cb720bc7f0dc5c06b not found: ID does not exist"
Sep 10 13:50:31.539275 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.539238    2578 scope.go:117] "RemoveContainer" containerID="4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6"
Sep 10 13:50:31.546984 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.546966    2578 scope.go:117] "RemoveContainer" containerID="4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6"
Sep 10 13:50:31.547305 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:50:31.547279    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6\": container with ID starting with 4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6 not found: ID does not exist" containerID="4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6"
Sep 10 13:50:31.547388 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.547311    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6"} err="failed to get container status \"4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6\": rpc error: code = NotFound desc = could not find container \"4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6\": container with ID starting with 4aa45ee8996268387aec5e59694d4340a319cb4eeb51f2665cddf8ace262c7a6 not found: ID does not exist"
Sep 10 13:50:31.547647 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.547628    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"]
Sep 10 13:50:31.551682 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.551661    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-645758-864958b789-lbv94"]
Sep 10 13:50:31.562813 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.562790    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"]
Sep 10 13:50:31.566987 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:31.566966    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-645758-6cc7f5d4bd-v4kc4"]
Sep 10 13:50:33.451369 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:33.451334    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfdc0310-c7da-421d-a3a9-da0c0c7f594c" path="/var/lib/kubelet/pods/bfdc0310-c7da-421d-a3a9-da0c0c7f594c/volumes"
Sep 10 13:50:33.451737 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:33.451704    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e25f62-8c4c-407a-b06a-09f6f63e0f35" path="/var/lib/kubelet/pods/e9e25f62-8c4c-407a-b06a-09f6f63e0f35/volumes"
Sep 10 13:50:48.556152 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.556118    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"]
Sep 10 13:50:48.556580 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.556364    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc" podUID="f439fcff-d8e2-419f-8a2c-b10b1a513422" containerName="maas-api" containerID="cri-o://c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c" gracePeriod=30
Sep 10 13:50:48.810892 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.810823    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:50:48.907862 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.907828    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-metrics-certs\") pod \"f439fcff-d8e2-419f-8a2c-b10b1a513422\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") "
Sep 10 13:50:48.907862 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.907871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f439fcff-d8e2-419f-8a2c-b10b1a513422-openshift-service-ca\") pod \"f439fcff-d8e2-419f-8a2c-b10b1a513422\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") "
Sep 10 13:50:48.908119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.907916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls\") pod \"f439fcff-d8e2-419f-8a2c-b10b1a513422\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") "
Sep 10 13:50:48.908119 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.907947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgpj7\" (UniqueName: \"kubernetes.io/projected/f439fcff-d8e2-419f-8a2c-b10b1a513422-kube-api-access-jgpj7\") pod \"f439fcff-d8e2-419f-8a2c-b10b1a513422\" (UID: \"f439fcff-d8e2-419f-8a2c-b10b1a513422\") "
Sep 10 13:50:48.908356 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.908323    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f439fcff-d8e2-419f-8a2c-b10b1a513422-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f439fcff-d8e2-419f-8a2c-b10b1a513422" (UID: "f439fcff-d8e2-419f-8a2c-b10b1a513422"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:50:48.910080 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.910056    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f439fcff-d8e2-419f-8a2c-b10b1a513422" (UID: "f439fcff-d8e2-419f-8a2c-b10b1a513422"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:48.910183 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.910076    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f439fcff-d8e2-419f-8a2c-b10b1a513422" (UID: "f439fcff-d8e2-419f-8a2c-b10b1a513422"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:50:48.910183 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:48.910154    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f439fcff-d8e2-419f-8a2c-b10b1a513422-kube-api-access-jgpj7" (OuterVolumeSpecName: "kube-api-access-jgpj7") pod "f439fcff-d8e2-419f-8a2c-b10b1a513422" (UID: "f439fcff-d8e2-419f-8a2c-b10b1a513422"). InnerVolumeSpecName "kube-api-access-jgpj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:50:49.009633 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.009597    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f439fcff-d8e2-419f-8a2c-b10b1a513422-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:49.009633 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.009632    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:49.009840 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.009646    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgpj7\" (UniqueName: \"kubernetes.io/projected/f439fcff-d8e2-419f-8a2c-b10b1a513422-kube-api-access-jgpj7\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:49.009840 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.009660    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f439fcff-d8e2-419f-8a2c-b10b1a513422-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:50:49.626446 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.626414    2578 generic.go:356] "Generic (PLEG): container finished" podID="f439fcff-d8e2-419f-8a2c-b10b1a513422" containerID="c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c" exitCode=0
Sep 10 13:50:49.626854 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.626462    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"
Sep 10 13:50:49.626854 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.626499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc" event={"ID":"f439fcff-d8e2-419f-8a2c-b10b1a513422","Type":"ContainerDied","Data":"c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c"}
Sep 10 13:50:49.626854 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.626563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc" event={"ID":"f439fcff-d8e2-419f-8a2c-b10b1a513422","Type":"ContainerDied","Data":"792bee1982bbd895fe4a0696e00fc1dcad51a1cca6b13fc77a850fe2d820f865"}
Sep 10 13:50:49.626854 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.626587    2578 scope.go:117] "RemoveContainer" containerID="c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c"
Sep 10 13:50:49.635516 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.635497    2578 scope.go:117] "RemoveContainer" containerID="c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c"
Sep 10 13:50:49.635838 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:50:49.635814    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c\": container with ID starting with c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c not found: ID does not exist" containerID="c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c"
Sep 10 13:50:49.635897 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.635847    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c"} err="failed to get container status \"c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c\": rpc error: code = NotFound desc = could not find container \"c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c\": container with ID starting with c0858eaab4f3005a28253cea7dfe1cb63f368126b164708961cf5d4bf9f45d5c not found: ID does not exist"
Sep 10 13:50:49.646351 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.646319    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"]
Sep 10 13:50:49.648105 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:49.648083    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-767dce-567d8bc89f-ftrqc"]
Sep 10 13:50:51.451603 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:50:51.451563    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f439fcff-d8e2-419f-8a2c-b10b1a513422" path="/var/lib/kubelet/pods/f439fcff-d8e2-419f-8a2c-b10b1a513422/volumes"
Sep 10 13:51:06.546176 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.546141    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"]
Sep 10 13:51:06.546808 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.546416    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc" podUID="9e497323-636b-46d2-8203-34c432db324c" containerName="maas-api" containerID="cri-o://5af82144c9e1100ca986459164e0b1afeb9dfe67717159fe0614e14fc6a81c90" gracePeriod=30
Sep 10 13:51:06.713397 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.713363    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e497323-636b-46d2-8203-34c432db324c" containerID="5af82144c9e1100ca986459164e0b1afeb9dfe67717159fe0614e14fc6a81c90" exitCode=0
Sep 10 13:51:06.713553 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.713440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc" event={"ID":"9e497323-636b-46d2-8203-34c432db324c","Type":"ContainerDied","Data":"5af82144c9e1100ca986459164e0b1afeb9dfe67717159fe0614e14fc6a81c90"}
Sep 10 13:51:06.791558 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.791529    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:51:06.873953 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.873850    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dndk7\" (UniqueName: \"kubernetes.io/projected/9e497323-636b-46d2-8203-34c432db324c-kube-api-access-dndk7\") pod \"9e497323-636b-46d2-8203-34c432db324c\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") "
Sep 10 13:51:06.873953 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.873903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e497323-636b-46d2-8203-34c432db324c-openshift-service-ca\") pod \"9e497323-636b-46d2-8203-34c432db324c\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") "
Sep 10 13:51:06.873953 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.873951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-maas-api-tls\") pod \"9e497323-636b-46d2-8203-34c432db324c\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") "
Sep 10 13:51:06.874263 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.873981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-metrics-certs\") pod \"9e497323-636b-46d2-8203-34c432db324c\" (UID: \"9e497323-636b-46d2-8203-34c432db324c\") "
Sep 10 13:51:06.874389 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.874362    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e497323-636b-46d2-8203-34c432db324c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e497323-636b-46d2-8203-34c432db324c" (UID: "9e497323-636b-46d2-8203-34c432db324c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:51:06.876165 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.876136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9e497323-636b-46d2-8203-34c432db324c" (UID: "9e497323-636b-46d2-8203-34c432db324c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:51:06.876165 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.876155    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e497323-636b-46d2-8203-34c432db324c" (UID: "9e497323-636b-46d2-8203-34c432db324c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:51:06.876330 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.876215    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e497323-636b-46d2-8203-34c432db324c-kube-api-access-dndk7" (OuterVolumeSpecName: "kube-api-access-dndk7") pod "9e497323-636b-46d2-8203-34c432db324c" (UID: "9e497323-636b-46d2-8203-34c432db324c"). InnerVolumeSpecName "kube-api-access-dndk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:51:06.974621 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.974584    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dndk7\" (UniqueName: \"kubernetes.io/projected/9e497323-636b-46d2-8203-34c432db324c-kube-api-access-dndk7\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:51:06.974621 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.974613    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e497323-636b-46d2-8203-34c432db324c-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:51:06.974621 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.974624    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:51:06.974621 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:06.974632    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e497323-636b-46d2-8203-34c432db324c-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:51:07.720802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:07.720695    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc" event={"ID":"9e497323-636b-46d2-8203-34c432db324c","Type":"ContainerDied","Data":"a4ff3dc46e4fd296cfc877029397226f309df4dbc03e3ae559d07b96fcbe3d41"}
Sep 10 13:51:07.720802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:07.720732    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"
Sep 10 13:51:07.720802 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:07.720757    2578 scope.go:117] "RemoveContainer" containerID="5af82144c9e1100ca986459164e0b1afeb9dfe67717159fe0614e14fc6a81c90"
Sep 10 13:51:07.741577 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:07.741547    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"]
Sep 10 13:51:07.744387 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:07.744359    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-89be92-69d5c447c-krqvc"]
Sep 10 13:51:09.451850 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:09.451816    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e497323-636b-46d2-8203-34c432db324c" path="/var/lib/kubelet/pods/9e497323-636b-46d2-8203-34c432db324c/volumes"
Sep 10 13:51:16.625249 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:16.625210    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-xpch5"]
Sep 10 13:51:16.763079 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:16.762993    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f57b54ff6-xpch5" podUID="f0012b8d-37b5-4964-a9cf-a845485ad693" containerName="manager" containerID="cri-o://fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579" gracePeriod=30
Sep 10 13:51:17.012702 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.012673    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:51:17.170307 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.170182    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq29p\" (UniqueName: \"kubernetes.io/projected/f0012b8d-37b5-4964-a9cf-a845485ad693-kube-api-access-mq29p\") pod \"f0012b8d-37b5-4964-a9cf-a845485ad693\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") "
Sep 10 13:51:17.170307 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.170253    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-metrics-certs\") pod \"f0012b8d-37b5-4964-a9cf-a845485ad693\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") "
Sep 10 13:51:17.170500 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.170448    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-webhook-cert\") pod \"f0012b8d-37b5-4964-a9cf-a845485ad693\" (UID: \"f0012b8d-37b5-4964-a9cf-a845485ad693\") "
Sep 10 13:51:17.172388 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.172354    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0012b8d-37b5-4964-a9cf-a845485ad693-kube-api-access-mq29p" (OuterVolumeSpecName: "kube-api-access-mq29p") pod "f0012b8d-37b5-4964-a9cf-a845485ad693" (UID: "f0012b8d-37b5-4964-a9cf-a845485ad693"). InnerVolumeSpecName "kube-api-access-mq29p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:51:17.172487 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.172388    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f0012b8d-37b5-4964-a9cf-a845485ad693" (UID: "f0012b8d-37b5-4964-a9cf-a845485ad693"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:51:17.172487 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.172368    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f0012b8d-37b5-4964-a9cf-a845485ad693" (UID: "f0012b8d-37b5-4964-a9cf-a845485ad693"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:51:17.271289 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.271245    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq29p\" (UniqueName: \"kubernetes.io/projected/f0012b8d-37b5-4964-a9cf-a845485ad693-kube-api-access-mq29p\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:51:17.271289 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.271281    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:51:17.271289 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.271294    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0012b8d-37b5-4964-a9cf-a845485ad693-webhook-cert\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:51:17.768214 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.768158    2578 generic.go:356] "Generic (PLEG): container finished" podID="f0012b8d-37b5-4964-a9cf-a845485ad693" containerID="fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579" exitCode=0
Sep 10 13:51:17.768602 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.768243    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-xpch5" event={"ID":"f0012b8d-37b5-4964-a9cf-a845485ad693","Type":"ContainerDied","Data":"fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579"}
Sep 10 13:51:17.768602 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.768327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-xpch5" event={"ID":"f0012b8d-37b5-4964-a9cf-a845485ad693","Type":"ContainerDied","Data":"f84b39b2a97d64efef92f51ade4d3ab8c7e48a63bba324bb1e37229be580fc8b"}
Sep 10 13:51:17.768602 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.768252    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-xpch5"
Sep 10 13:51:17.768602 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.768349    2578 scope.go:117] "RemoveContainer" containerID="fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579"
Sep 10 13:51:17.777665 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.777645    2578 scope.go:117] "RemoveContainer" containerID="fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579"
Sep 10 13:51:17.777921 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:51:17.777899    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579\": container with ID starting with fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579 not found: ID does not exist" containerID="fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579"
Sep 10 13:51:17.777974 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.777930    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579"} err="failed to get container status \"fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579\": rpc error: code = NotFound desc = could not find container \"fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579\": container with ID starting with fff8336a16005d8efbaa3c1f83181aad161d55bf22ea8dc17de4deac3f6aa579 not found: ID does not exist"
Sep 10 13:51:17.791768 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.791744    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-xpch5"]
Sep 10 13:51:17.793985 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:17.793966    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-xpch5"]
Sep 10 13:51:19.451563 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:51:19.451531    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0012b8d-37b5-4964-a9cf-a845485ad693" path="/var/lib/kubelet/pods/f0012b8d-37b5-4964-a9cf-a845485ad693/volumes"
Sep 10 13:52:18.663550 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663510    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-kzmsh"]
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663892    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e497323-636b-46d2-8203-34c432db324c" containerName="maas-api"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663908    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e497323-636b-46d2-8203-34c432db324c" containerName="maas-api"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663918    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" containerName="maas-api"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663924    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" containerName="maas-api"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663931    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7106e4a-e027-4e84-923d-0c554292f85c" containerName="maas-api"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663936    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7106e4a-e027-4e84-923d-0c554292f85c" containerName="maas-api"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663944    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81269b8b-8532-494f-be96-afe0e17ed5ff" containerName="payload-pre-processing"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663949    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81269b8b-8532-494f-be96-afe0e17ed5ff" containerName="payload-pre-processing"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663961    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e25f62-8c4c-407a-b06a-09f6f63e0f35" containerName="payload-processing"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663966    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e25f62-8c4c-407a-b06a-09f6f63e0f35" containerName="payload-processing"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663977    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdc0310-c7da-421d-a3a9-da0c0c7f594c" containerName="maas-api"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663986    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdc0310-c7da-421d-a3a9-da0c0c7f594c" containerName="maas-api"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.663995    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f439fcff-d8e2-419f-8a2c-b10b1a513422" containerName="maas-api"
Sep 10 13:52:18.663988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664000    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f439fcff-d8e2-419f-8a2c-b10b1a513422" containerName="maas-api"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664006    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2582fe27-9967-4282-be6e-b4f70af2f9fe" containerName="payload-processing"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664011    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2582fe27-9967-4282-be6e-b4f70af2f9fe" containerName="payload-processing"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664020    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0012b8d-37b5-4964-a9cf-a845485ad693" containerName="manager"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664026    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0012b8d-37b5-4964-a9cf-a845485ad693" containerName="manager"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664083    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a29dea3-3b4b-44ca-b2b6-44cc9d7e5762" containerName="maas-api"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664092    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdc0310-c7da-421d-a3a9-da0c0c7f594c" containerName="maas-api"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664106    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e25f62-8c4c-407a-b06a-09f6f63e0f35" containerName="payload-processing"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664115    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f439fcff-d8e2-419f-8a2c-b10b1a513422" containerName="maas-api"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664124    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0012b8d-37b5-4964-a9cf-a845485ad693" containerName="manager"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664133    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="81269b8b-8532-494f-be96-afe0e17ed5ff" containerName="payload-pre-processing"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664143    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7106e4a-e027-4e84-923d-0c554292f85c" containerName="maas-api"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664152    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e497323-636b-46d2-8203-34c432db324c" containerName="maas-api"
Sep 10 13:52:18.664738 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.664159    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2582fe27-9967-4282-be6e-b4f70af2f9fe" containerName="payload-processing"
Sep 10 13:52:18.668355 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.668338    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.671936 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.671913    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 13:52:18.672154 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.672136    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vtl9v\""
Sep 10 13:52:18.672251 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.672175    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 13:52:18.672251 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.672177    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 13:52:18.678879 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.678856    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-kzmsh"]
Sep 10 13:52:18.799878 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.799824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8dj\" (UniqueName: \"kubernetes.io/projected/c002dbfe-874c-404f-9055-3729cf57c24c-kube-api-access-bc8dj\") pod \"maas-controller-6f57b54ff6-kzmsh\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") " pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.800070 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.799899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-webhook-cert\") pod \"maas-controller-6f57b54ff6-kzmsh\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") " pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.800070 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.800020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-metrics-certs\") pod \"maas-controller-6f57b54ff6-kzmsh\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") " pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.900756 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.900719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-metrics-certs\") pod \"maas-controller-6f57b54ff6-kzmsh\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") " pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.900933 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.900762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8dj\" (UniqueName: \"kubernetes.io/projected/c002dbfe-874c-404f-9055-3729cf57c24c-kube-api-access-bc8dj\") pod \"maas-controller-6f57b54ff6-kzmsh\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") " pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.900933 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.900796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-webhook-cert\") pod \"maas-controller-6f57b54ff6-kzmsh\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") " pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.903225 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.903174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-webhook-cert\") pod \"maas-controller-6f57b54ff6-kzmsh\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") " pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.903355 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.903238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-metrics-certs\") pod \"maas-controller-6f57b54ff6-kzmsh\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") " pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.911360 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.911338    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8dj\" (UniqueName: \"kubernetes.io/projected/c002dbfe-874c-404f-9055-3729cf57c24c-kube-api-access-bc8dj\") pod \"maas-controller-6f57b54ff6-kzmsh\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") " pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:18.979116 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:18.979019    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:19.254003 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:19.253978    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-kzmsh"]
Sep 10 13:52:19.256651 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:52:19.256627    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc002dbfe_874c_404f_9055_3729cf57c24c.slice/crio-567c2d2a1b0d24f5768913ce26746b2702ab651db2274e91b261a3bbd78a285b WatchSource:0}: Error finding container 567c2d2a1b0d24f5768913ce26746b2702ab651db2274e91b261a3bbd78a285b: Status 404 returned error can't find the container with id 567c2d2a1b0d24f5768913ce26746b2702ab651db2274e91b261a3bbd78a285b
Sep 10 13:52:19.258070 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:19.258052    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 13:52:20.064948 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:20.064849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh" event={"ID":"c002dbfe-874c-404f-9055-3729cf57c24c","Type":"ContainerStarted","Data":"5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63"}
Sep 10 13:52:20.065359 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:20.064987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:20.065359 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:20.065030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh" event={"ID":"c002dbfe-874c-404f-9055-3729cf57c24c","Type":"ContainerStarted","Data":"567c2d2a1b0d24f5768913ce26746b2702ab651db2274e91b261a3bbd78a285b"}
Sep 10 13:52:20.104459 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:20.104402    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh" podStartSLOduration=1.593170794 podStartE2EDuration="2.104383135s" totalImagesPullingTime="511.212341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:20.102949009 +0000 UTC m=+2045.119780982" watchObservedRunningTime="2026-09-10 13:52:20.104383135 +0000 UTC m=+2045.121215108"
Sep 10 13:52:31.072478 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:31.072446    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:40.470970 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.470865    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-kzmsh"]
Sep 10 13:52:40.471449 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.471421    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh" podUID="c002dbfe-874c-404f-9055-3729cf57c24c" containerName="manager" containerID="cri-o://5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63" gracePeriod=30
Sep 10 13:52:40.722933 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.722866    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:40.782767 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.782731    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc8dj\" (UniqueName: \"kubernetes.io/projected/c002dbfe-874c-404f-9055-3729cf57c24c-kube-api-access-bc8dj\") pod \"c002dbfe-874c-404f-9055-3729cf57c24c\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") "
Sep 10 13:52:40.782972 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.782806    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-webhook-cert\") pod \"c002dbfe-874c-404f-9055-3729cf57c24c\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") "
Sep 10 13:52:40.782972 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.782874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-metrics-certs\") pod \"c002dbfe-874c-404f-9055-3729cf57c24c\" (UID: \"c002dbfe-874c-404f-9055-3729cf57c24c\") "
Sep 10 13:52:40.785002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.784967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c002dbfe-874c-404f-9055-3729cf57c24c-kube-api-access-bc8dj" (OuterVolumeSpecName: "kube-api-access-bc8dj") pod "c002dbfe-874c-404f-9055-3729cf57c24c" (UID: "c002dbfe-874c-404f-9055-3729cf57c24c"). InnerVolumeSpecName "kube-api-access-bc8dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:52:40.785121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.784984    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c002dbfe-874c-404f-9055-3729cf57c24c" (UID: "c002dbfe-874c-404f-9055-3729cf57c24c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:52:40.785121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.785030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c002dbfe-874c-404f-9055-3729cf57c24c" (UID: "c002dbfe-874c-404f-9055-3729cf57c24c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:52:40.884336 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.884293    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc8dj\" (UniqueName: \"kubernetes.io/projected/c002dbfe-874c-404f-9055-3729cf57c24c-kube-api-access-bc8dj\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:52:40.884336 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.884329    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-webhook-cert\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:52:40.884336 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:40.884338    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c002dbfe-874c-404f-9055-3729cf57c24c-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:52:41.173687 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.173651    2578 generic.go:356] "Generic (PLEG): container finished" podID="c002dbfe-874c-404f-9055-3729cf57c24c" containerID="5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63" exitCode=0
Sep 10 13:52:41.173964 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.173702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh" event={"ID":"c002dbfe-874c-404f-9055-3729cf57c24c","Type":"ContainerDied","Data":"5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63"}
Sep 10 13:52:41.173964 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.173712    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh"
Sep 10 13:52:41.173964 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.173771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-kzmsh" event={"ID":"c002dbfe-874c-404f-9055-3729cf57c24c","Type":"ContainerDied","Data":"567c2d2a1b0d24f5768913ce26746b2702ab651db2274e91b261a3bbd78a285b"}
Sep 10 13:52:41.173964 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.173794    2578 scope.go:117] "RemoveContainer" containerID="5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63"
Sep 10 13:52:41.183934 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.183914    2578 scope.go:117] "RemoveContainer" containerID="5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63"
Sep 10 13:52:41.184288 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:52:41.184266    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63\": container with ID starting with 5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63 not found: ID does not exist" containerID="5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63"
Sep 10 13:52:41.184349 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.184297    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63"} err="failed to get container status \"5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63\": rpc error: code = NotFound desc = could not find container \"5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63\": container with ID starting with 5add4da5e461beea2715bee83b4f91ff67d1b96bcbc343062e706a917a9e4c63 not found: ID does not exist"
Sep 10 13:52:41.197180 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.197145    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-kzmsh"]
Sep 10 13:52:41.201590 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.201558    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-kzmsh"]
Sep 10 13:52:41.452002 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:41.451923    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c002dbfe-874c-404f-9055-3729cf57c24c" path="/var/lib/kubelet/pods/c002dbfe-874c-404f-9055-3729cf57c24c/volumes"
Sep 10 13:52:42.119909 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.119876    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-8hhr8"]
Sep 10 13:52:42.120295 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.120248    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c002dbfe-874c-404f-9055-3729cf57c24c" containerName="manager"
Sep 10 13:52:42.120295 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.120260    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c002dbfe-874c-404f-9055-3729cf57c24c" containerName="manager"
Sep 10 13:52:42.120369 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.120340    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c002dbfe-874c-404f-9055-3729cf57c24c" containerName="manager"
Sep 10 13:52:42.124706 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.124685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.127336 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.127314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 13:52:42.128224 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.128185    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 13:52:42.128298 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.128185    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 13:52:42.128355 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.128294    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vtl9v\""
Sep 10 13:52:42.132390 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.132363    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-8hhr8"]
Sep 10 13:52:42.194858 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.194816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75p9\" (UniqueName: \"kubernetes.io/projected/e26c9725-6101-420b-831d-238b414e1c9e-kube-api-access-c75p9\") pod \"maas-controller-6f57b54ff6-8hhr8\" (UID: \"e26c9725-6101-420b-831d-238b414e1c9e\") " pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.195034 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.194922    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26c9725-6101-420b-831d-238b414e1c9e-metrics-certs\") pod \"maas-controller-6f57b54ff6-8hhr8\" (UID: \"e26c9725-6101-420b-831d-238b414e1c9e\") " pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.195034 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.194948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e26c9725-6101-420b-831d-238b414e1c9e-webhook-cert\") pod \"maas-controller-6f57b54ff6-8hhr8\" (UID: \"e26c9725-6101-420b-831d-238b414e1c9e\") " pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.295993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.295953    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26c9725-6101-420b-831d-238b414e1c9e-metrics-certs\") pod \"maas-controller-6f57b54ff6-8hhr8\" (UID: \"e26c9725-6101-420b-831d-238b414e1c9e\") " pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.295993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.295994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e26c9725-6101-420b-831d-238b414e1c9e-webhook-cert\") pod \"maas-controller-6f57b54ff6-8hhr8\" (UID: \"e26c9725-6101-420b-831d-238b414e1c9e\") " pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.296175 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.296048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75p9\" (UniqueName: \"kubernetes.io/projected/e26c9725-6101-420b-831d-238b414e1c9e-kube-api-access-c75p9\") pod \"maas-controller-6f57b54ff6-8hhr8\" (UID: \"e26c9725-6101-420b-831d-238b414e1c9e\") " pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.298596 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.298567    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e26c9725-6101-420b-831d-238b414e1c9e-webhook-cert\") pod \"maas-controller-6f57b54ff6-8hhr8\" (UID: \"e26c9725-6101-420b-831d-238b414e1c9e\") " pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.298719 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.298695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e26c9725-6101-420b-831d-238b414e1c9e-metrics-certs\") pod \"maas-controller-6f57b54ff6-8hhr8\" (UID: \"e26c9725-6101-420b-831d-238b414e1c9e\") " pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.307236 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.307171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75p9\" (UniqueName: \"kubernetes.io/projected/e26c9725-6101-420b-831d-238b414e1c9e-kube-api-access-c75p9\") pod \"maas-controller-6f57b54ff6-8hhr8\" (UID: \"e26c9725-6101-420b-831d-238b414e1c9e\") " pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.436178 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.436090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:42.707018 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:42.706990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f57b54ff6-8hhr8"]
Sep 10 13:52:42.709402 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:52:42.709375    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26c9725_6101_420b_831d_238b414e1c9e.slice/crio-3a2500c678c46a17202e7b804148e81b32cff6c97ebca65c894e47df3536bdc8 WatchSource:0}: Error finding container 3a2500c678c46a17202e7b804148e81b32cff6c97ebca65c894e47df3536bdc8: Status 404 returned error can't find the container with id 3a2500c678c46a17202e7b804148e81b32cff6c97ebca65c894e47df3536bdc8
Sep 10 13:52:43.185654 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:43.185618    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-8hhr8" event={"ID":"e26c9725-6101-420b-831d-238b414e1c9e","Type":"ContainerStarted","Data":"3a2500c678c46a17202e7b804148e81b32cff6c97ebca65c894e47df3536bdc8"}
Sep 10 13:52:44.191646 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:44.191613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f57b54ff6-8hhr8" event={"ID":"e26c9725-6101-420b-831d-238b414e1c9e","Type":"ContainerStarted","Data":"4647714cde7000b088dc19563b9ad84f053adea6661e3fe250cd2b8c22e2c3fa"}
Sep 10 13:52:44.192046 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:44.191673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:52:44.210008 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:44.209954    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f57b54ff6-8hhr8" podStartSLOduration=1.5726933939999999 podStartE2EDuration="2.209938293s" totalImagesPullingTime="637.244899ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:52:44.209043227 +0000 UTC m=+2069.225875199" watchObservedRunningTime="2026-09-10 13:52:44.209938293 +0000 UTC m=+2069.226770582"
Sep 10 13:52:55.200105 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:52:55.200072    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f57b54ff6-8hhr8"
Sep 10 13:53:55.068645 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.068608    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"]
Sep 10 13:53:55.069116 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.068856    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96" podUID="9708ec74-1e0e-4253-9566-48e2ba024d5b" containerName="manager" containerID="cri-o://9944758f1a1be65440ab5d2dae3aa7af3cfeb85d6f44738b0112e2cdc7b14386" gracePeriod=10
Sep 10 13:53:55.539860 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.539831    2578 generic.go:356] "Generic (PLEG): container finished" podID="9708ec74-1e0e-4253-9566-48e2ba024d5b" containerID="9944758f1a1be65440ab5d2dae3aa7af3cfeb85d6f44738b0112e2cdc7b14386" exitCode=0
Sep 10 13:53:55.539993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.539887    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96" event={"ID":"9708ec74-1e0e-4253-9566-48e2ba024d5b","Type":"ContainerDied","Data":"9944758f1a1be65440ab5d2dae3aa7af3cfeb85d6f44738b0112e2cdc7b14386"}
Sep 10 13:53:55.622416 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.622380    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:53:55.719506 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.719419    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jspw2\" (UniqueName: \"kubernetes.io/projected/9708ec74-1e0e-4253-9566-48e2ba024d5b-kube-api-access-jspw2\") pod \"9708ec74-1e0e-4253-9566-48e2ba024d5b\" (UID: \"9708ec74-1e0e-4253-9566-48e2ba024d5b\") "
Sep 10 13:53:55.719506 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.719505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9708ec74-1e0e-4253-9566-48e2ba024d5b-extensions-socket-volume\") pod \"9708ec74-1e0e-4253-9566-48e2ba024d5b\" (UID: \"9708ec74-1e0e-4253-9566-48e2ba024d5b\") "
Sep 10 13:53:55.719893 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.719865    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9708ec74-1e0e-4253-9566-48e2ba024d5b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9708ec74-1e0e-4253-9566-48e2ba024d5b" (UID: "9708ec74-1e0e-4253-9566-48e2ba024d5b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:53:55.721455 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.721433    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9708ec74-1e0e-4253-9566-48e2ba024d5b-kube-api-access-jspw2" (OuterVolumeSpecName: "kube-api-access-jspw2") pod "9708ec74-1e0e-4253-9566-48e2ba024d5b" (UID: "9708ec74-1e0e-4253-9566-48e2ba024d5b"). InnerVolumeSpecName "kube-api-access-jspw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:53:55.821068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.821018    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jspw2\" (UniqueName: \"kubernetes.io/projected/9708ec74-1e0e-4253-9566-48e2ba024d5b-kube-api-access-jspw2\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:53:55.821068 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:55.821063    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9708ec74-1e0e-4253-9566-48e2ba024d5b-extensions-socket-volume\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:53:56.546452 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:56.546415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96" event={"ID":"9708ec74-1e0e-4253-9566-48e2ba024d5b","Type":"ContainerDied","Data":"126efb475468e97a14a74e7f3486d7398e9ffafca8343f4602fb8f16c9a07976"}
Sep 10 13:53:56.546452 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:56.546436    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"
Sep 10 13:53:56.546925 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:56.546487    2578 scope.go:117] "RemoveContainer" containerID="9944758f1a1be65440ab5d2dae3aa7af3cfeb85d6f44738b0112e2cdc7b14386"
Sep 10 13:53:56.569815 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:56.569779    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"]
Sep 10 13:53:56.574128 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:56.574098    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5hp96"]
Sep 10 13:53:57.451873 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:53:57.451834    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9708ec74-1e0e-4253-9566-48e2ba024d5b" path="/var/lib/kubelet/pods/9708ec74-1e0e-4253-9566-48e2ba024d5b/volumes"
Sep 10 13:54:55.773009 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.772970    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"]
Sep 10 13:54:55.773510 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.773377    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9708ec74-1e0e-4253-9566-48e2ba024d5b" containerName="manager"
Sep 10 13:54:55.773510 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.773389    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9708ec74-1e0e-4253-9566-48e2ba024d5b" containerName="manager"
Sep 10 13:54:55.773510 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.773442    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9708ec74-1e0e-4253-9566-48e2ba024d5b" containerName="manager"
Sep 10 13:54:55.776402 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.776386    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:54:55.779231 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.779212    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6svwj\""
Sep 10 13:54:55.790332 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.790311    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"]
Sep 10 13:54:55.839046 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.839011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/650d2e97-1faa-40ae-a4f7-997c9ecf447e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-njv6d\" (UID: \"650d2e97-1faa-40ae-a4f7-997c9ecf447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:54:55.839046 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.839048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7bb7\" (UniqueName: \"kubernetes.io/projected/650d2e97-1faa-40ae-a4f7-997c9ecf447e-kube-api-access-g7bb7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-njv6d\" (UID: \"650d2e97-1faa-40ae-a4f7-997c9ecf447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:54:55.939636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.939601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/650d2e97-1faa-40ae-a4f7-997c9ecf447e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-njv6d\" (UID: \"650d2e97-1faa-40ae-a4f7-997c9ecf447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:54:55.939636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.939639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7bb7\" (UniqueName: \"kubernetes.io/projected/650d2e97-1faa-40ae-a4f7-997c9ecf447e-kube-api-access-g7bb7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-njv6d\" (UID: \"650d2e97-1faa-40ae-a4f7-997c9ecf447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:54:55.940009 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.939989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/650d2e97-1faa-40ae-a4f7-997c9ecf447e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-njv6d\" (UID: \"650d2e97-1faa-40ae-a4f7-997c9ecf447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:54:55.952026 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:55.951991    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7bb7\" (UniqueName: \"kubernetes.io/projected/650d2e97-1faa-40ae-a4f7-997c9ecf447e-kube-api-access-g7bb7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-njv6d\" (UID: \"650d2e97-1faa-40ae-a4f7-997c9ecf447e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:54:56.086551 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:56.086452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:54:56.363342 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:56.363268    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"]
Sep 10 13:54:56.367404 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:54:56.367375    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650d2e97_1faa_40ae_a4f7_997c9ecf447e.slice/crio-076f59e3147fc943895afd25fb7a20b93957a8735f2ecce79e67dbcb5f64e19b WatchSource:0}: Error finding container 076f59e3147fc943895afd25fb7a20b93957a8735f2ecce79e67dbcb5f64e19b: Status 404 returned error can't find the container with id 076f59e3147fc943895afd25fb7a20b93957a8735f2ecce79e67dbcb5f64e19b
Sep 10 13:54:56.846868 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:56.846834    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d" event={"ID":"650d2e97-1faa-40ae-a4f7-997c9ecf447e","Type":"ContainerStarted","Data":"ae108d282eb8aac04653de0a549f78c6d0f6bc3659107206d18d010a4c818ae6"}
Sep 10 13:54:56.847276 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:56.846886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d" event={"ID":"650d2e97-1faa-40ae-a4f7-997c9ecf447e","Type":"ContainerStarted","Data":"076f59e3147fc943895afd25fb7a20b93957a8735f2ecce79e67dbcb5f64e19b"}
Sep 10 13:54:56.847276 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:56.846931    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:54:56.870473 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:54:56.870412    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d" podStartSLOduration=1.8703897 podStartE2EDuration="1.8703897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:54:56.868360214 +0000 UTC m=+2201.885192185" watchObservedRunningTime="2026-09-10 13:54:56.8703897 +0000 UTC m=+2201.887221674"
Sep 10 13:55:07.854282 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:07.854246    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-njv6d"
Sep 10 13:55:44.862264 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.862153    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"]
Sep 10 13:55:44.866031 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.866012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:44.868668 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.868648    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e9da5e77-kserve-self-signed-certs\""
Sep 10 13:55:44.872867 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.872826    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"]
Sep 10 13:55:44.967189 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.967141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tmp-dir\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:44.967405 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.967229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-dshm\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:44.967405 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.967289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-model-cache\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:44.967405 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.967349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tls-certs\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:44.967405 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.967386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-972n8\" (UniqueName: \"kubernetes.io/projected/8807fd5e-ea7b-47db-aa5f-27d580e2286d-kube-api-access-972n8\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:44.967551 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:44.967436    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-home\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.068516 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.068477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tmp-dir\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.068516 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.068523    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-dshm\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.068784 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.068560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-model-cache\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.068784 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.068610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tls-certs\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.068784 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.068633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-972n8\" (UniqueName: \"kubernetes.io/projected/8807fd5e-ea7b-47db-aa5f-27d580e2286d-kube-api-access-972n8\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.068784 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.068662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-home\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.069021 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.068939    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tmp-dir\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.069021 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.068994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-model-cache\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.069021 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.069015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-home\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.070891 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.070867    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-dshm\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.071287 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.071261    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tls-certs\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.080420 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.080390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-972n8\" (UniqueName: \"kubernetes.io/projected/8807fd5e-ea7b-47db-aa5f-27d580e2286d-kube-api-access-972n8\") pod \"e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") " pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.176889 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.176800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:45.454816 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:45.454792    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"]
Sep 10 13:55:45.456313 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:55:45.456286    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8807fd5e_ea7b_47db_aa5f_27d580e2286d.slice/crio-338e1c65ea63483cd77bc2d4156c9fdb19db19f1dd89eae320888c258bd2500e WatchSource:0}: Error finding container 338e1c65ea63483cd77bc2d4156c9fdb19db19f1dd89eae320888c258bd2500e: Status 404 returned error can't find the container with id 338e1c65ea63483cd77bc2d4156c9fdb19db19f1dd89eae320888c258bd2500e
Sep 10 13:55:46.082242 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:46.082180    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l" event={"ID":"8807fd5e-ea7b-47db-aa5f-27d580e2286d","Type":"ContainerStarted","Data":"14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8"}
Sep 10 13:55:46.082626 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:46.082256    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l" event={"ID":"8807fd5e-ea7b-47db-aa5f-27d580e2286d","Type":"ContainerStarted","Data":"338e1c65ea63483cd77bc2d4156c9fdb19db19f1dd89eae320888c258bd2500e"}
Sep 10 13:55:46.101993 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:46.101928    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l" podStartSLOduration=2.101906948 podStartE2EDuration="2.101906948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:55:46.099687621 +0000 UTC m=+2251.116519593" watchObservedRunningTime="2026-09-10 13:55:46.101906948 +0000 UTC m=+2251.118738928"
Sep 10 13:55:55.177089 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:55.177040    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:55.177622 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:55.177108    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:55.189811 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:55.189784    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:55:56.144908 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:55:56.144873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:56:03.481129 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:03.481091    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"]
Sep 10 13:56:18.501234 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.501156    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l" podUID="8807fd5e-ea7b-47db-aa5f-27d580e2286d" containerName="main" containerID="cri-o://14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8" gracePeriod=45
Sep 10 13:56:18.754065 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.753999    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:56:18.868079 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868040    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-model-cache\") pod \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") "
Sep 10 13:56:18.868257 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-home\") pod \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") "
Sep 10 13:56:18.868257 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868131    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-972n8\" (UniqueName: \"kubernetes.io/projected/8807fd5e-ea7b-47db-aa5f-27d580e2286d-kube-api-access-972n8\") pod \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") "
Sep 10 13:56:18.868377 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868258    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-dshm\") pod \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") "
Sep 10 13:56:18.868377 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868299    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tmp-dir\") pod \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") "
Sep 10 13:56:18.868377 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868342    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8807fd5e-ea7b-47db-aa5f-27d580e2286d" (UID: "8807fd5e-ea7b-47db-aa5f-27d580e2286d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:56:18.868377 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868364    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tls-certs\") pod \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\" (UID: \"8807fd5e-ea7b-47db-aa5f-27d580e2286d\") "
Sep 10 13:56:18.868596 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868387    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-home" (OuterVolumeSpecName: "home") pod "8807fd5e-ea7b-47db-aa5f-27d580e2286d" (UID: "8807fd5e-ea7b-47db-aa5f-27d580e2286d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:56:18.868596 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868530    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8807fd5e-ea7b-47db-aa5f-27d580e2286d" (UID: "8807fd5e-ea7b-47db-aa5f-27d580e2286d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:56:18.868720 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868609    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tmp-dir\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:18.868720 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868624    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-model-cache\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:18.868720 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.868633    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-home\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:18.870371 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.870347    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-dshm" (OuterVolumeSpecName: "dshm") pod "8807fd5e-ea7b-47db-aa5f-27d580e2286d" (UID: "8807fd5e-ea7b-47db-aa5f-27d580e2286d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 13:56:18.870465 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.870445    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8807fd5e-ea7b-47db-aa5f-27d580e2286d" (UID: "8807fd5e-ea7b-47db-aa5f-27d580e2286d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:56:18.870521 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.870501    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8807fd5e-ea7b-47db-aa5f-27d580e2286d-kube-api-access-972n8" (OuterVolumeSpecName: "kube-api-access-972n8") pod "8807fd5e-ea7b-47db-aa5f-27d580e2286d" (UID: "8807fd5e-ea7b-47db-aa5f-27d580e2286d"). InnerVolumeSpecName "kube-api-access-972n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:56:18.969081 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.969041    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-972n8\" (UniqueName: \"kubernetes.io/projected/8807fd5e-ea7b-47db-aa5f-27d580e2286d-kube-api-access-972n8\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:18.969081 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.969071    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8807fd5e-ea7b-47db-aa5f-27d580e2286d-dshm\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:18.969081 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:18.969082    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8807fd5e-ea7b-47db-aa5f-27d580e2286d-tls-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:19.254294 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.254260    2578 generic.go:356] "Generic (PLEG): container finished" podID="8807fd5e-ea7b-47db-aa5f-27d580e2286d" containerID="14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8" exitCode=0
Sep 10 13:56:19.254477 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.254313    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"
Sep 10 13:56:19.254477 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.254327    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l" event={"ID":"8807fd5e-ea7b-47db-aa5f-27d580e2286d","Type":"ContainerDied","Data":"14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8"}
Sep 10 13:56:19.254477 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.254357    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l" event={"ID":"8807fd5e-ea7b-47db-aa5f-27d580e2286d","Type":"ContainerDied","Data":"338e1c65ea63483cd77bc2d4156c9fdb19db19f1dd89eae320888c258bd2500e"}
Sep 10 13:56:19.254477 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.254375    2578 scope.go:117] "RemoveContainer" containerID="14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8"
Sep 10 13:56:19.263265 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.263244    2578 scope.go:117] "RemoveContainer" containerID="14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8"
Sep 10 13:56:19.263575 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:56:19.263553    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8\": container with ID starting with 14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8 not found: ID does not exist" containerID="14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8"
Sep 10 13:56:19.263644 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.263582    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8"} err="failed to get container status \"14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8\": rpc error: code = NotFound desc = could not find container \"14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8\": container with ID starting with 14bbc6ff2ca2119758bf93030586011db1ba5fddf4772fcffb007c261bb6dfc8 not found: ID does not exist"
Sep 10 13:56:19.276415 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.276384    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"]
Sep 10 13:56:19.283391 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.283360    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e9da5e77-kserve-78d8b6bbb7-qcv7l"]
Sep 10 13:56:19.451963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:19.451924    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8807fd5e-ea7b-47db-aa5f-27d580e2286d" path="/var/lib/kubelet/pods/8807fd5e-ea7b-47db-aa5f-27d580e2286d/volumes"
Sep 10 13:56:24.397758 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.397723    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"]
Sep 10 13:56:24.398135 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.398088    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8807fd5e-ea7b-47db-aa5f-27d580e2286d" containerName="main"
Sep 10 13:56:24.398135 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.398099    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8807fd5e-ea7b-47db-aa5f-27d580e2286d" containerName="main"
Sep 10 13:56:24.398224 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.398176    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8807fd5e-ea7b-47db-aa5f-27d580e2286d" containerName="main"
Sep 10 13:56:24.400958 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.400936    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.411701 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.411672    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"]
Sep 10 13:56:24.417131 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.417095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbb9d\" (UniqueName: \"kubernetes.io/projected/f1a2518c-753b-4491-9da3-5a8bec689e32-kube-api-access-zbb9d\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.417284 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.417176    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1a2518c-753b-4491-9da3-5a8bec689e32-metrics-certs\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.417284 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.417279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a2518c-753b-4491-9da3-5a8bec689e32-maas-api-tls\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.417370 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.417305    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a2518c-753b-4491-9da3-5a8bec689e32-openshift-service-ca\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.517691 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.517655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbb9d\" (UniqueName: \"kubernetes.io/projected/f1a2518c-753b-4491-9da3-5a8bec689e32-kube-api-access-zbb9d\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.517891 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.517726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1a2518c-753b-4491-9da3-5a8bec689e32-metrics-certs\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.517891 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.517780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a2518c-753b-4491-9da3-5a8bec689e32-maas-api-tls\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.517891 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.517802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a2518c-753b-4491-9da3-5a8bec689e32-openshift-service-ca\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.518491 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.518467    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1a2518c-753b-4491-9da3-5a8bec689e32-openshift-service-ca\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.520260 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.520239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1a2518c-753b-4491-9da3-5a8bec689e32-metrics-certs\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.520345 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.520283    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1a2518c-753b-4491-9da3-5a8bec689e32-maas-api-tls\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.526215 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.526176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbb9d\" (UniqueName: \"kubernetes.io/projected/f1a2518c-753b-4491-9da3-5a8bec689e32-kube-api-access-zbb9d\") pod \"maas-api-dbdb6b6cd-f92tg\" (UID: \"f1a2518c-753b-4491-9da3-5a8bec689e32\") " pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.715386 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.715284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:24.995583 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:24.995556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"]
Sep 10 13:56:24.997417 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:56:24.997386    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a2518c_753b_4491_9da3_5a8bec689e32.slice/crio-344ecd362df5d69906cbedebca13a0c4686fc8b0890c3618750a53862e47bb1e WatchSource:0}: Error finding container 344ecd362df5d69906cbedebca13a0c4686fc8b0890c3618750a53862e47bb1e: Status 404 returned error can't find the container with id 344ecd362df5d69906cbedebca13a0c4686fc8b0890c3618750a53862e47bb1e
Sep 10 13:56:25.284662 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:25.284572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg" event={"ID":"f1a2518c-753b-4491-9da3-5a8bec689e32","Type":"ContainerStarted","Data":"344ecd362df5d69906cbedebca13a0c4686fc8b0890c3618750a53862e47bb1e"}
Sep 10 13:56:26.293578 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:26.293521    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg" event={"ID":"f1a2518c-753b-4491-9da3-5a8bec689e32","Type":"ContainerStarted","Data":"8df917844788eb93d3fa562155518aff0123804e5136f3cfa4875ef74af3b0e4"}
Sep 10 13:56:26.293985 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:26.293601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:26.314286 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:26.314231    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg" podStartSLOduration=1.792133736 podStartE2EDuration="2.314214029s" totalImagesPullingTime="522.080293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 13:56:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:56:26.312536638 +0000 UTC m=+2291.329368610" watchObservedRunningTime="2026-09-10 13:56:26.314214029 +0000 UTC m=+2291.331046002"
Sep 10 13:56:32.305481 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:32.305447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-dbdb6b6cd-f92tg"
Sep 10 13:56:40.322050 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:40.322005    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"]
Sep 10 13:56:40.322749 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:40.322709    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b" podUID="68191959-d911-4c6a-8cae-8d65cbad9d1b" containerName="maas-api" containerID="cri-o://f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315" gracePeriod=30
Sep 10 13:56:40.413707 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:40.413662    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b" podUID="68191959-d911-4c6a-8cae-8d65cbad9d1b" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.34:8443/health\": dial tcp 10.134.0.34:8443: connect: connection refused"
Sep 10 13:56:41.571366 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.571338    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:56:41.666145 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.666044    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phjbg\" (UniqueName: \"kubernetes.io/projected/68191959-d911-4c6a-8cae-8d65cbad9d1b-kube-api-access-phjbg\") pod \"68191959-d911-4c6a-8cae-8d65cbad9d1b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") "
Sep 10 13:56:41.666145 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.666089    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-maas-api-tls\") pod \"68191959-d911-4c6a-8cae-8d65cbad9d1b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") "
Sep 10 13:56:41.666145 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.666120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68191959-d911-4c6a-8cae-8d65cbad9d1b-openshift-service-ca\") pod \"68191959-d911-4c6a-8cae-8d65cbad9d1b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") "
Sep 10 13:56:41.666472 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.666254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-metrics-certs\") pod \"68191959-d911-4c6a-8cae-8d65cbad9d1b\" (UID: \"68191959-d911-4c6a-8cae-8d65cbad9d1b\") "
Sep 10 13:56:41.666567 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.666542    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68191959-d911-4c6a-8cae-8d65cbad9d1b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68191959-d911-4c6a-8cae-8d65cbad9d1b" (UID: "68191959-d911-4c6a-8cae-8d65cbad9d1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 13:56:41.668207 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.668170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68191959-d911-4c6a-8cae-8d65cbad9d1b" (UID: "68191959-d911-4c6a-8cae-8d65cbad9d1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:56:41.668316 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.668231    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68191959-d911-4c6a-8cae-8d65cbad9d1b" (UID: "68191959-d911-4c6a-8cae-8d65cbad9d1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 13:56:41.668316 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.668244    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68191959-d911-4c6a-8cae-8d65cbad9d1b-kube-api-access-phjbg" (OuterVolumeSpecName: "kube-api-access-phjbg") pod "68191959-d911-4c6a-8cae-8d65cbad9d1b" (UID: "68191959-d911-4c6a-8cae-8d65cbad9d1b"). InnerVolumeSpecName "kube-api-access-phjbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 13:56:41.767657 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.767621    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-metrics-certs\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:41.767657 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.767652    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phjbg\" (UniqueName: \"kubernetes.io/projected/68191959-d911-4c6a-8cae-8d65cbad9d1b-kube-api-access-phjbg\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:41.767657 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.767662    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68191959-d911-4c6a-8cae-8d65cbad9d1b-maas-api-tls\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:41.767903 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:41.767674    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68191959-d911-4c6a-8cae-8d65cbad9d1b-openshift-service-ca\") on node \"ip-10-0-138-19.ec2.internal\" DevicePath \"\""
Sep 10 13:56:42.377990 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:42.377948    2578 generic.go:356] "Generic (PLEG): container finished" podID="68191959-d911-4c6a-8cae-8d65cbad9d1b" containerID="f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315" exitCode=0
Sep 10 13:56:42.378146 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:42.378006    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"
Sep 10 13:56:42.378146 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:42.378018    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b" event={"ID":"68191959-d911-4c6a-8cae-8d65cbad9d1b","Type":"ContainerDied","Data":"f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315"}
Sep 10 13:56:42.378146 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:42.378088    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b" event={"ID":"68191959-d911-4c6a-8cae-8d65cbad9d1b","Type":"ContainerDied","Data":"5bbfb67030acf771605bb96a1b4f3d77d62ef401073fc487d6d5f02e41940d48"}
Sep 10 13:56:42.378146 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:42.378111    2578 scope.go:117] "RemoveContainer" containerID="f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315"
Sep 10 13:56:42.387221 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:42.387184    2578 scope.go:117] "RemoveContainer" containerID="f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315"
Sep 10 13:56:42.387506 ip-10-0-138-19 kubenswrapper[2578]: E0910 13:56:42.387490    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315\": container with ID starting with f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315 not found: ID does not exist" containerID="f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315"
Sep 10 13:56:42.387561 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:42.387514    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315"} err="failed to get container status \"f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315\": rpc error: code = NotFound desc = could not find container \"f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315\": container with ID starting with f9aae88f47e71a73feb71654da5b37af98ef4488bcb5eff1dad012d7dae8f315 not found: ID does not exist"
Sep 10 13:56:42.401458 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:42.401425    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"]
Sep 10 13:56:42.406013 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:42.405984    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b9dddc78b-s426b"]
Sep 10 13:56:43.451122 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:43.451087    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68191959-d911-4c6a-8cae-8d65cbad9d1b" path="/var/lib/kubelet/pods/68191959-d911-4c6a-8cae-8d65cbad9d1b/volumes"
Sep 10 13:56:59.538247 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:59.538212    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4x8vz_19ac0b83-57a5-4e97-b2f0-6f5535a27b80/manager/0.log"
Sep 10 13:56:59.661563 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:59.661528    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q2hl2_235f19de-7f94-40ef-9e2e-d4edd3ed63b6/manager/0.log"
Sep 10 13:56:59.788758 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:56:59.788672    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f57b54ff6-8hhr8_e26c9725-6101-420b-831d-238b414e1c9e/manager/0.log"
Sep 10 13:57:00.140983 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:00.140904    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d444f55-2wmkd_17e155fc-d390-49c3-9741-bc94d0b88df7/copy-manifests/0.log"
Sep 10 13:57:00.148692 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:00.148663    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d444f55-2wmkd_17e155fc-d390-49c3-9741-bc94d0b88df7/manager/0.log"
Sep 10 13:57:00.728754 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:00.728724    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-dbdb6b6cd-f92tg_f1a2518c-753b-4491-9da3-5a8bec689e32/maas-api/0.log"
Sep 10 13:57:01.058400 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:01.058369    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-jdcxn_95f8e0d2-6c06-4cb5-812d-2fd07963ad2c/postgres/0.log"
Sep 10 13:57:01.945945 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:01.945913    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-j7drr_6c1a6add-5be2-448d-8b0f-7b8823bba5b9/manager/0.log"
Sep 10 13:57:02.062857 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:02.062820    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-qcnjp_f0bbb328-8ba6-4850-a81c-983074807cc6/manager/0.log"
Sep 10 13:57:02.173090 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:02.173064    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xhdvm_9dea981c-a6d9-42ed-af66-599fed284de7/kuadrant-console-plugin/0.log"
Sep 10 13:57:02.298780 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:02.298747    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-njv6d_650d2e97-1faa-40ae-a4f7-997c9ecf447e/manager/0.log"
Sep 10 13:57:02.549308 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:02.549223    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-psqmm_2677b59e-dacf-4ed4-a4dd-36ac8095e2ee/manager/0.log"
Sep 10 13:57:03.131311 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:03.131277    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb8b5959f-85mgm_7143a86a-20f1-44c9-83be-528e37661a57/kube-auth-proxy/0.log"
Sep 10 13:57:03.626027 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:03.625989    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cs9hb_2114350c-4310-43c2-9fc9-4ef47e717a06/payload-processing/0.log"
Sep 10 13:57:04.218818 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:04.218790    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-f7fmf_4b433723-19a3-4a10-b3a1-b5da19098a06/main/0.log"
Sep 10 13:57:04.682710 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:04.682679    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5zdpt_654f76b4-cb45-4048-af3c-cd33c2e79fb2/main/0.log"
Sep 10 13:57:13.716029 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:13.715956    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-62tzd_c9e7452e-8ac7-40b2-a170-fb36f5f4cff9/global-pull-secret-syncer/0.log"
Sep 10 13:57:13.996133 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:13.996091    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z52mw_8fbe2e10-4ae2-4914-9e54-fa46bdc66b94/konnectivity-agent/0.log"
Sep 10 13:57:14.052378 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:14.052336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-19.ec2.internal_102348c45e238ec4cfbabd8b1ecec91f/haproxy/0.log"
Sep 10 13:57:18.575166 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:18.575136    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-j7drr_6c1a6add-5be2-448d-8b0f-7b8823bba5b9/manager/0.log"
Sep 10 13:57:18.627159 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:18.627130    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-qcnjp_f0bbb328-8ba6-4850-a81c-983074807cc6/manager/0.log"
Sep 10 13:57:18.666289 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:18.666257    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xhdvm_9dea981c-a6d9-42ed-af66-599fed284de7/kuadrant-console-plugin/0.log"
Sep 10 13:57:18.720687 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:18.720651    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-njv6d_650d2e97-1faa-40ae-a4f7-997c9ecf447e/manager/0.log"
Sep 10 13:57:18.835793 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:18.835696    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-psqmm_2677b59e-dacf-4ed4-a4dd-36ac8095e2ee/manager/0.log"
Sep 10 13:57:20.630113 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:20.630084    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-h827l_e8842d11-4ed0-401f-9cd4-9479765035a3/kube-state-metrics/0.log"
Sep 10 13:57:20.650526 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:20.650489    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-h827l_e8842d11-4ed0-401f-9cd4-9479765035a3/kube-rbac-proxy-main/0.log"
Sep 10 13:57:20.675902 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:20.675869    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-h827l_e8842d11-4ed0-401f-9cd4-9479765035a3/kube-rbac-proxy-self/0.log"
Sep 10 13:57:21.029810 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:21.029770    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-srmm7_c1130036-b991-491a-b56c-4717d21e4023/node-exporter/0.log"
Sep 10 13:57:21.060115 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:21.060088    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-srmm7_c1130036-b991-491a-b56c-4717d21e4023/kube-rbac-proxy/0.log"
Sep 10 13:57:21.098662 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:21.098633    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-srmm7_c1130036-b991-491a-b56c-4717d21e4023/init-textfile/0.log"
Sep 10 13:57:21.249546 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:21.249516    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-zm9th_89a8fe67-1f1e-423b-ab4b-0465f9bbcb64/kube-rbac-proxy-main/0.log"
Sep 10 13:57:21.274272 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:21.274245    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-zm9th_89a8fe67-1f1e-423b-ab4b-0465f9bbcb64/kube-rbac-proxy-self/0.log"
Sep 10 13:57:21.300018 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:21.299989    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-zm9th_89a8fe67-1f1e-423b-ab4b-0465f9bbcb64/openshift-state-metrics/0.log"
Sep 10 13:57:21.536464 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:21.536376    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-2fdjx_05797d67-a8e9-418a-807a-8f6b521fa0f1/prometheus-operator/0.log"
Sep 10 13:57:21.559064 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:21.559034    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-2fdjx_05797d67-a8e9-418a-807a-8f6b521fa0f1/kube-rbac-proxy/0.log"
Sep 10 13:57:21.586150 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:21.586119    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-85hmm_f87cd7ce-82ce-4272-b867-47107fbaa79d/prometheus-operator-admission-webhook/0.log"
Sep 10 13:57:23.818623 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.818587    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"]
Sep 10 13:57:23.819145 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.819125    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68191959-d911-4c6a-8cae-8d65cbad9d1b" containerName="maas-api"
Sep 10 13:57:23.819259 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.819149    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="68191959-d911-4c6a-8cae-8d65cbad9d1b" containerName="maas-api"
Sep 10 13:57:23.819259 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.819248    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="68191959-d911-4c6a-8cae-8d65cbad9d1b" containerName="maas-api"
Sep 10 13:57:23.822121 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.822105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.824464 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.824443    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n2zxf\"/\"kube-root-ca.crt\""
Sep 10 13:57:23.824630 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.824610    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n2zxf\"/\"openshift-service-ca.crt\""
Sep 10 13:57:23.825481 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.825467    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n2zxf\"/\"default-dockercfg-frlhp\""
Sep 10 13:57:23.830263 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.830236    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"]
Sep 10 13:57:23.831636 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.831611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-lib-modules\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.831728 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.831661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-proc\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.831728 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.831691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-podres\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.831728 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.831718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6fgr\" (UniqueName: \"kubernetes.io/projected/e5f46dcd-7193-448a-98f6-ad5471a6f572-kube-api-access-d6fgr\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.831906 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.831734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-sys\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.932605 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.932566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-proc\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.932808 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.932619    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-podres\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.932808 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.932648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fgr\" (UniqueName: \"kubernetes.io/projected/e5f46dcd-7193-448a-98f6-ad5471a6f572-kube-api-access-d6fgr\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.932808 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.932671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-sys\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.932808 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.932747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-lib-modules\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.933003 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.932695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-proc\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.938677 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.933033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-lib-modules\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.938677 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.933209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-podres\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.938677 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.933313    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5f46dcd-7193-448a-98f6-ad5471a6f572-sys\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:23.943033 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:23.942994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6fgr\" (UniqueName: \"kubernetes.io/projected/e5f46dcd-7193-448a-98f6-ad5471a6f572-kube-api-access-d6fgr\") pod \"perf-node-gather-daemonset-crs8p\" (UID: \"e5f46dcd-7193-448a-98f6-ad5471a6f572\") " pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:24.132364 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:24.132271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:24.403436 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:24.403410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"]
Sep 10 13:57:24.405666 ip-10-0-138-19 kubenswrapper[2578]: W0910 13:57:24.405638    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode5f46dcd_7193_448a_98f6_ad5471a6f572.slice/crio-9f46413d3d6f175501deca5f0faf0bee1010f4baeebf17cf6a0906673c959485 WatchSource:0}: Error finding container 9f46413d3d6f175501deca5f0faf0bee1010f4baeebf17cf6a0906673c959485: Status 404 returned error can't find the container with id 9f46413d3d6f175501deca5f0faf0bee1010f4baeebf17cf6a0906673c959485
Sep 10 13:57:24.581230 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:24.581176    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p" event={"ID":"e5f46dcd-7193-448a-98f6-ad5471a6f572","Type":"ContainerStarted","Data":"9c1f1ca8e87321d41683f8e50dc92b63496ecf9a41f6af8e204fef1450fa4d3f"}
Sep 10 13:57:24.581411 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:24.581249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p" event={"ID":"e5f46dcd-7193-448a-98f6-ad5471a6f572","Type":"ContainerStarted","Data":"9f46413d3d6f175501deca5f0faf0bee1010f4baeebf17cf6a0906673c959485"}
Sep 10 13:57:24.581411 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:24.581287    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:24.602441 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:24.602391    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p" podStartSLOduration=1.6023747670000001 podStartE2EDuration="1.602374767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 13:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 13:57:24.600492926 +0000 UTC m=+2349.617324920" watchObservedRunningTime="2026-09-10 13:57:24.602374767 +0000 UTC m=+2349.619206787"
Sep 10 13:57:26.385732 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:26.385693    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ch54g_63ac7b44-01ac-4815-8cb8-fa1af93a57f7/dns/0.log"
Sep 10 13:57:26.412051 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:26.412022    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ch54g_63ac7b44-01ac-4815-8cb8-fa1af93a57f7/kube-rbac-proxy/0.log"
Sep 10 13:57:26.500963 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:26.500939    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vjgdt_2e22079e-1953-486a-8fca-835b0eb1092f/dns-node-resolver/0.log"
Sep 10 13:57:27.049085 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:27.049049    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-576868dd66-clmnw_36f9a8c0-7e20-40de-b964-0ba36aab5ea0/registry/0.log"
Sep 10 13:57:27.072356 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:27.072330    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-64j9z_28616fa8-74fb-434f-8238-4246116c85b2/node-ca/0.log"
Sep 10 13:57:28.358627 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:28.358596    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb8b5959f-85mgm_7143a86a-20f1-44c9-83be-528e37661a57/kube-auth-proxy/0.log"
Sep 10 13:57:28.524725 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:28.524675    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cs9hb_2114350c-4310-43c2-9fc9-4ef47e717a06/payload-processing/0.log"
Sep 10 13:57:29.433838 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:29.433803    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6976s_14d54b58-eea2-4440-b09f-6445e29204e4/serve-healthcheck-canary/0.log"
Sep 10 13:57:30.097123 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:30.097094    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rnmmb_d2dd98c8-d5d5-422a-a507-1480a9609412/kube-rbac-proxy/0.log"
Sep 10 13:57:30.120478 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:30.120450    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rnmmb_d2dd98c8-d5d5-422a-a507-1480a9609412/exporter/0.log"
Sep 10 13:57:30.146627 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:30.146601    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rnmmb_d2dd98c8-d5d5-422a-a507-1480a9609412/extractor/0.log"
Sep 10 13:57:30.596431 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:30.596403    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n2zxf/perf-node-gather-daemonset-crs8p"
Sep 10 13:57:32.428474 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:32.428444    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-4x8vz_19ac0b83-57a5-4e97-b2f0-6f5535a27b80/manager/0.log"
Sep 10 13:57:32.753792 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:32.753756    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q2hl2_235f19de-7f94-40ef-9e2e-d4edd3ed63b6/manager/0.log"
Sep 10 13:57:32.821072 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:32.821039    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f57b54ff6-8hhr8_e26c9725-6101-420b-831d-238b414e1c9e/manager/0.log"
Sep 10 13:57:32.951619 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:32.951579    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d444f55-2wmkd_17e155fc-d390-49c3-9741-bc94d0b88df7/manager/0.log"
Sep 10 13:57:32.974711 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:32.974683    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84d444f55-2wmkd_17e155fc-d390-49c3-9741-bc94d0b88df7/copy-manifests/0.log"
Sep 10 13:57:34.420988 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:34.420959    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb8b5959f-85mgm_7143a86a-20f1-44c9-83be-528e37661a57/kube-auth-proxy/0.log"
Sep 10 13:57:34.545771 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:34.545735    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cs9hb_2114350c-4310-43c2-9fc9-4ef47e717a06/payload-processing/0.log"
Sep 10 13:57:36.271891 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:36.271853    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb8b5959f-85mgm_7143a86a-20f1-44c9-83be-528e37661a57/kube-auth-proxy/0.log"
Sep 10 13:57:36.405765 ip-10-0-138-19 kubenswrapper[2578]: I0910 13:57:36.405733    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-cs9hb_2114350c-4310-43c2-9fc9-4ef47e717a06/payload-processing/0.log"