Sep 06 10:21:47.202605 ip-10-0-130-109 systemd[1]: Starting Kubernetes Kubelet...
Sep 06 10:21:47.620145 ip-10-0-130-109 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 10:21:47.620145 ip-10-0-130-109 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 06 10:21:47.620145 ip-10-0-130-109 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 10:21:47.620145 ip-10-0-130-109 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 06 10:21:47.622319 ip-10-0-130-109 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 10:21:47.622583 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.622457    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 06 10:21:47.627059 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627041    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:47.627059 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627059    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:47.627059 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627062    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627066    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627070    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627073    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627078    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627080    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627083    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627086    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627089    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627091    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627094    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627096    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627099    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627103    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627107    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627110    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627113    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627116    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627119    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:47.627159 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627122    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627140    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627144    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627147    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627150    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627153    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627155    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627158    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627161    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627164    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627167    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627170    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627173    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627176    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627179    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627181    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627184    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627187    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627189    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627192    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:47.627660 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627194    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627197    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627199    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627202    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627204    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627207    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627209    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627212    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627214    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627217    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627220    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627223    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627226    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627229    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627232    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627235    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627237    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627240    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627244    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:47.628173 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627248    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627250    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627253    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627256    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627259    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627262    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627277    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627280    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627283    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627286    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627289    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627292    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627295    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627297    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627300    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627302    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627305    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627308    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627310    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:47.628657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627313    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627316    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627319    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627322    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627325    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627327    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627330    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627334    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627337    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627340    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627343    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627345    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627348    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627351    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627353    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627356    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627359    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627361    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627364    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:47.629119 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.627368    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:47.629691 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629676    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:47.629691 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629690    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629694    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629698    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629701    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629704    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629707    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629711    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629716    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629719    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629722    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629725    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629728    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629731    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629735    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629738    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629741    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629743    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629746    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629749    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:47.629749 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629751    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629755    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629758    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629761    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629764    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629767    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629769    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629772    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629774    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629777    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629779    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629782    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629785    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629787    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629790    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629792    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629795    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629798    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629800    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:47.630291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629803    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629805    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629808    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629810    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629813    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629815    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629817    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629820    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629824    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629826    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629829    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629831    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629834    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629836    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629839    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629842    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629845    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629848    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629850    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629853    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:47.630763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629858    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629862    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629864    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629868    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629871    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629874    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629877    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629881    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629883    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629886    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629889    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629892    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629895    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629897    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629900    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629903    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629905    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629908    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629911    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629913    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:47.631280 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629915    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629919    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629922    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629924    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629927    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629929    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629932    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629935    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629939    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629942    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629946    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629949    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629952    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629955    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629957    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629960    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629962    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629965    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629967    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.629970    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:47.631769 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630401    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630406    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630409    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630412    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630415    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630417    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630420    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630422    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630425    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630428    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630431    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630452    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630456    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630459    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630463    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630466    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630469    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630472    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630477    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630480    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:47.632291 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630483    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630486    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630488    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630491    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630494    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630497    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630499    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630502    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630506    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630510    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630514    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630517    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630520    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630523    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630526    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630529    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630531    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630534    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630536    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:47.632881 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630539    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630542    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630545    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630547    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630550    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630553    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630555    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630558    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630561    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630564    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630566    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630569    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630572    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630575    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630578    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630581    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630583    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630586    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630588    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630591    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:47.633378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630593    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630596    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630598    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630600    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630603    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630605    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630608    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630610    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630612    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630615    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630619    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630622    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630625    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630627    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630630    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630634    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630637    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630639    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630644    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:47.633878 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630646    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630650    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630653    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630656    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630659    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630662    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630665    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630667    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630670    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630672    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630675    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630677    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630680    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630682    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630685    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630687    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630690    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630692    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630695    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630697    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:47.634367 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.630700    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630779    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630813    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630820    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630825    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630829    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630832    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630836    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630841    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630844    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630847    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630851    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630854    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630857    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630862    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630866    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630869    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630872    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630875    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630879    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630882    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630885    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630889    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630892    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 06 10:21:47.634871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630896    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630899    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630902    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630905    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630908    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630911    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630914    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630917    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630920    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630925    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630928    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630931    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630934    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630937    2583 flags.go:64] FLAG: --enable-server="true"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630940    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630945    2583 flags.go:64] FLAG: --event-burst="100"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630948    2583 flags.go:64] FLAG: --event-qps="50"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630951    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630954    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630957    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630962    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630964    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630967    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630971    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630974    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 06 10:21:47.635473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630977    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630980    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630983    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630986    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630989    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630992    2583 flags.go:64] FLAG: --feature-gates=""
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630995    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.630998    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631001    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631004    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631007    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631010    2583 flags.go:64] FLAG: --help="false"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631013    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-130-109.ec2.internal"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631016    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631019    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631023    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631028    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631034    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631038    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631042    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631046    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631051    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631056    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631061    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 06 10:21:47.636129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631064    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631067    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631070    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631074    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631077    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631080    2583 flags.go:64] FLAG: --lock-file=""
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631082    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631087    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631090    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631096    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631099    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631102    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631105    2583 flags.go:64] FLAG: --logging-format="text"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631108    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631111    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631114    2583 flags.go:64] FLAG: --manifest-url=""
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631117    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631122    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631125    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631129    2583 flags.go:64] FLAG: --max-pods="110"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631132    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631135    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631138    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631141    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631144    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 06 10:21:47.636762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631147    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631150    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631157    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631160    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631163    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631167    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631169    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631175    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631178    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631181    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631184    2583 flags.go:64] FLAG: --port="10250"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631187    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631190    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0050ec57e3de134fa"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631193    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631196    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631200    2583 flags.go:64] FLAG: --register-node="true"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631203    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631207    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631209    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631212    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631215    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631219    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631222    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631225    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631228    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 06 10:21:47.637452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631231    2583 flags.go:64] FLAG: --runonce="false"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631234    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631237    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631240    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631243    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631246    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631249    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631252    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631255    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631258    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631261    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631277    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631281    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631284    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631288    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631291    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631296    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631299    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631302    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631306    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631309    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631312    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631315    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631318    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631321    2583 flags.go:64] FLAG: --v="2"
Sep 06 10:21:47.638140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631326    2583 flags.go:64] FLAG: --version="false"
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631330    2583 flags.go:64] FLAG: --vmodule=""
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631334    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631337    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631466    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631471    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631474    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631477    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631480    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631482    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631487    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631490    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631492    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631495    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631497    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631500    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631503    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631506    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631508    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631511    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:47.638783 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631513    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631516    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631519    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631521    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631524    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631526    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631529    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631531    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631536    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631539    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631541    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631545    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631547    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631550    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631553    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631555    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631558    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631561    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631563    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631566    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:47.639292 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631569    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631571    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631576    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631579    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631581    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631584    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631586    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631589    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631591    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631594    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631597    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631599    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631602    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631604    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631607    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631609    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631612    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631615    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631617    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:47.639824 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631620    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631624    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631626    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631629    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631632    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631635    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631638    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631640    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631643    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631645    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631648    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631651    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631653    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631656    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631658    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631662    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631665    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631667    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631670    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:47.640324 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631672    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631676    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631680    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631683    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631685    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631688    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631691    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631693    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631696    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631698    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631701    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631703    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631706    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631710    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631714    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631717    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631720    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631723    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631725    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:47.640808 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631728    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631731    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631733    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631736    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631739    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.631741    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.631746    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.639541    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.639561    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.640521    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.640530    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640590    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640595    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640598    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640601    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:47.641286 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640604    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640607    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640609    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640612    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640615    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640618    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640620    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640623    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640626    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640629    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640632    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640634    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640637    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640639    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640642    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640644    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640647    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640649    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640652    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640655    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:47.641688 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640657    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640660    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640664    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640668    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640672    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640676    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640679    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640681    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640684    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640687    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640689    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640692    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640694    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640697    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640699    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640702    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640705    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640707    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640710    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:47.642189 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640713    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640715    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640718    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640721    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640723    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640726    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640728    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640730    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640733    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640736    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640738    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640741    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640743    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640746    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640748    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640751    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640753    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640756    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640759    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640763    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:47.642754 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640766    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640768    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640771    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640773    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640776    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640778    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640781    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640783    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640786    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640789    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640791    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640795    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640798    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640801    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640804    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640806    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640808    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640811    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640814    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640817    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:47.643296 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640820    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640822    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640825    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640828    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640830    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640833    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640835    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640838    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640841    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640844    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640847    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640850    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640853    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640857    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640860    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:47.643811 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640863    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.640869    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.640996    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641001    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641004    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641007    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641010    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641012    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641015    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641017    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641020    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641023    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641027    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641030    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641033    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:21:47.644205 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641035    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641038    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641041    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641043    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641046    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641049    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641051    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641054    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641056    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641059    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641061    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641064    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641067    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641069    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641072    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641075    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641078    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641080    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641083    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641085    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:21:47.644606 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641088    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641091    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641093    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641096    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641098    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641100    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641103    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641105    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641108    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641110    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641113    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641116    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641118    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641120    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641123    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641125    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641128    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641130    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641134    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641137    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:21:47.645161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641140    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641143    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641146    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641148    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641151    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641154    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641157    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641160    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641163    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641165    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641168    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641170    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641173    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641176    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641178    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641181    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641183    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641186    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641188    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:21:47.645723 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641190    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641193    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641195    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641198    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641201    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641203    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641206    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641208    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641211    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641213    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641216    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641218    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641221    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641223    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641226    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641228    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641231    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641234    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641236    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641239    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:21:47.646207 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641241    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641245    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641249    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641252    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641255    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641258    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:47.641261    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.641278    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.641412    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.643790    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.644912    2583 server.go:1019] "Starting client certificate rotation"
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.645009    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 10:21:47.646726 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.645051    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 10:21:47.648217 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.648202    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 10:21:47.655010 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.654984    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 10:21:47.670458 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.670427    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 06 10:21:47.673946 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.673928    2583 log.go:25] "Validated CRI v1 image API"
Sep 06 10:21:47.674930 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.674912    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 06 10:21:47.678609 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.678591    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 10:21:47.681320 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.681301    2583 fs.go:135] Filesystem UUIDs: map[362b28a6-2bef-4e8b-a13f-d6c33570fad1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 86e3efd1-b8e0-491b-93d1-11ae2793cc39:/dev/nvme0n1p3]
Sep 06 10:21:47.681379 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.681320    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 06 10:21:47.685696 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.685568    2583 manager.go:216] Machine: {Timestamp:2026-09-06 10:21:47.685063589 +0000 UTC m=+0.372772439 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499996 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b5fa31db46db8c057128fcba03193 SystemUUID:ec2b5fa3-1db4-6db8-c057-128fcba03193 BootID:3f424563-f588-46b4-89b5-8b777f5e15c9 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f5:11:04:2d:e5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f5:11:04:2d:e5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9e:ff:c6:04:42:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 06 10:21:47.685696 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.685688    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 06 10:21:47.685809 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.685783    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 06 10:21:47.687718 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.687682    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 06 10:21:47.687906 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.687724    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-109.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 06 10:21:47.688630 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.688617    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 06 10:21:47.688696 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.688635    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 06 10:21:47.688696 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.688646    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 10:21:47.689623 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.689609    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 10:21:47.691461 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.691447    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 06 10:21:47.695479 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.695461    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 10:21:47.695642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.695630    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 06 10:21:47.695705 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.695673    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 06 10:21:47.695705 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.695696    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 06 10:21:47.695796 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.695714    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 06 10:21:47.695796 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.695730    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 06 10:21:47.695796 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.695743    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 06 10:21:47.696155 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.696141    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 10:21:47.696223 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.696165    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 10:21:47.699587 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.699566    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 06 10:21:47.702384 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.702370    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 06 10:21:47.702433 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.702400    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 06 10:21:47.702567 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.702557    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 06 10:21:47.702598 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.702572    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 06 10:21:47.702598 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.702578    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 06 10:21:47.702598 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.702583    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 06 10:21:47.702598 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.702590    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 06 10:21:47.702598 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.702595    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 06 10:21:47.703240 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.703227    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 06 10:21:47.703287 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.703242    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 06 10:21:47.703287 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.703254    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 06 10:21:47.703287 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.703262    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 06 10:21:47.703374 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.703289    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 06 10:21:47.703374 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.703301    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 06 10:21:47.704224 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.704207    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 06 10:21:47.704430 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.704417    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 06 10:21:47.706748 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.706667    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 06 10:21:47.706863 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.706773    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 06 10:21:47.709666 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.709646    2583 server.go:1268] "Started kubelet"
Sep 06 10:21:47.709781 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.709722    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 06 10:21:47.709897 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.709794    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 06 10:21:47.709897 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.709868    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 06 10:21:47.710843 ip-10-0-130-109 systemd[1]: Started Kubernetes Kubelet.
Sep 06 10:21:47.712312 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.712298    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 06 10:21:47.712656 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.712641    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 06 10:21:47.715750 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.715723    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 06 10:21:47.716995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.716973    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-109.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 06 10:21:47.717711 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.717693    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 06 10:21:47.717784 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.717710    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 06 10:21:47.718200 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.717006    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-109.ec2.internal.18d2b42d1349464e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-109.ec2.internal,UID:ip-10-0-130-109.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-109.ec2.internal,},FirstTimestamp:2026-09-06 10:21:47.709605454 +0000 UTC m=+0.397314303,LastTimestamp:2026-09-06 10:21:47.709605454 +0000 UTC m=+0.397314303,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-109.ec2.internal,}"
Sep 06 10:21:47.718389 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.718373    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:47.718816 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.718799    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 06 10:21:47.718891 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.718831    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 06 10:21:47.718891 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.718843    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 06 10:21:47.718979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.718973    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 06 10:21:47.719016 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.718981    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 06 10:21:47.720881 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.720865    2583 factory.go:55] Registering systemd factory
Sep 06 10:21:47.721018 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.721007    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 06 10:21:47.721409 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.721373    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-109.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 06 10:21:47.721479 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.721407    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 06 10:21:47.721567 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.721548    2583 factory.go:153] Registering CRI-O factory
Sep 06 10:21:47.721567 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.721566    2583 factory.go:223] Registration of the crio container factory successfully
Sep 06 10:21:47.721687 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.721613    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 06 10:21:47.721687 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.721633    2583 factory.go:103] Registering Raw factory
Sep 06 10:21:47.721687 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.721649    2583 manager.go:1196] Started watching for new ooms in manager
Sep 06 10:21:47.722035 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.722025    2583 manager.go:349] Starting recovery of all containers
Sep 06 10:21:47.731816 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.731754    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-csssd"
Sep 06 10:21:47.732899 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.732875    2583 manager.go:354] Recovery completed
Sep 06 10:21:47.738885 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.738862    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:47.739564 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.739544    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-csssd"
Sep 06 10:21:47.740842 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.740822    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:47.740931 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.740860    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:47.740931 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.740875    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:47.741517 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.741502    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 06 10:21:47.741517 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.741514    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:47.741666 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.741516    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 06 10:21:47.741666 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.741558    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 10:21:47.742277 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.742255    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:47.742329 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.742294    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:47.742329 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.742305    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:47.742440 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.742425    2583 policy_none.go:49] "None policy: Start"
Sep 06 10:21:47.742484 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.742446    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 06 10:21:47.742484 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.742459    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 06 10:21:47.743304 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.743292    2583 policy_none.go:47] "Start"
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.785970    2583 manager.go:337] "Starting Device Plugin manager"
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.786010    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.786020    2583 server.go:87] "Starting device plugin registration server"
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.786342    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.786354    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.786452    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.786535    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.786543    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.787437    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 06 10:21:47.792487 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.787489    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:47.817151 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.817105    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 06 10:21:47.818482 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.818459    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 06 10:21:47.818587 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.818524    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 06 10:21:47.818587 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.818552    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 06 10:21:47.818675 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.818597    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 06 10:21:47.821156 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.821137    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:47.887432 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.887334    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:47.888394 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.888377    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:47.888460 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.888408    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:47.888460 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.888418    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:47.888460 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.888445    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-109.ec2.internal"
Sep 06 10:21:47.898724 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.898697    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-109.ec2.internal"
Sep 06 10:21:47.898838 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.898727    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-109.ec2.internal\": node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:47.919330 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.919297    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal"]
Sep 06 10:21:47.919449 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.919375    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:47.920825 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.920806    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:47.920932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.920836    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:47.920932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.920846    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:47.922999 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.922985    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:47.923134 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.923118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:47.923195 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.923155    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:47.923777 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.923762    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:47.923862 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.923797    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:47.923862 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.923812    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:47.923862 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.923765    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:47.923862 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.923861    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:47.924041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.923871    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:47.924234 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.924216    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:47.925976 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.925961    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:47.926088 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.925988    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:21:47.926768 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.926752    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:21:47.926852 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.926780    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:21:47.926852 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:47.926789    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:21:47.955136 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.955111    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-109.ec2.internal\" not found" node="ip-10-0-130-109.ec2.internal"
Sep 06 10:21:47.959696 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:47.959677    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-109.ec2.internal\" not found" node="ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.020206 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:48.020179    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61346eed96f8fec99ad05cd0344256fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal\" (UID: \"61346eed96f8fec99ad05cd0344256fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.020361 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:48.020212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/caa56ac7ebc4ca9eb8489a6688c0f5a0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-109.ec2.internal\" (UID: \"caa56ac7ebc4ca9eb8489a6688c0f5a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.020361 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:48.020228    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61346eed96f8fec99ad05cd0344256fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal\" (UID: \"61346eed96f8fec99ad05cd0344256fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.025315 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:48.025296    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:48.121148 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:48.121113    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61346eed96f8fec99ad05cd0344256fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal\" (UID: \"61346eed96f8fec99ad05cd0344256fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.121148 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:48.121145    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61346eed96f8fec99ad05cd0344256fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal\" (UID: \"61346eed96f8fec99ad05cd0344256fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.121299 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:48.121161    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/caa56ac7ebc4ca9eb8489a6688c0f5a0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-109.ec2.internal\" (UID: \"caa56ac7ebc4ca9eb8489a6688c0f5a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.121299 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:48.121197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/61346eed96f8fec99ad05cd0344256fa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal\" (UID: \"61346eed96f8fec99ad05cd0344256fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.121299 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:48.121202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/61346eed96f8fec99ad05cd0344256fa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal\" (UID: \"61346eed96f8fec99ad05cd0344256fa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.121299 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:48.121234    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/caa56ac7ebc4ca9eb8489a6688c0f5a0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-109.ec2.internal\" (UID: \"caa56ac7ebc4ca9eb8489a6688c0f5a0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:48.126235 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:48.126208    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:49.642164 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:49.642130    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:49.672347 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:49.672290    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:49.677623 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:49.677401    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:49.743225 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:49.743182    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:49.843699 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:49.843666    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:49.944312 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:49.944246    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:50.040382 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.040354    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:50.044919 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:50.044901    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:50.060080 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.060060    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 06 10:21:50.060222 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.060203    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 10:21:50.060316 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.060227    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 10:21:50.144987 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:50.144958    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:50.156738 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.156685    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 10:16:49 +0000 UTC" deadline="2028-05-23 07:35:24.974694434 +0000 UTC"
Sep 06 10:21:50.156738 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.156734    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14997h13m34.817963796s"
Sep 06 10:21:50.245538 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:50.245511    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:50.346120 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:50.346087    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-109.ec2.internal\" not found"
Sep 06 10:21:50.369388 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.369217    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:50.377438 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:50.377397    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa56ac7ebc4ca9eb8489a6688c0f5a0.slice/crio-56821d7f648d7586256a65ffb8957e8cda12ff8bc1748fdd285603de0b4f2bea WatchSource:0}: Error finding container 56821d7f648d7586256a65ffb8957e8cda12ff8bc1748fdd285603de0b4f2bea: Status 404 returned error can't find the container with id 56821d7f648d7586256a65ffb8957e8cda12ff8bc1748fdd285603de0b4f2bea
Sep 06 10:21:50.377684 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:50.377664    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61346eed96f8fec99ad05cd0344256fa.slice/crio-adea93393b2119572382b063d5ec5d82da0f3f1f6a3b33c582dc5fea4575fda1 WatchSource:0}: Error finding container adea93393b2119572382b063d5ec5d82da0f3f1f6a3b33c582dc5fea4575fda1: Status 404 returned error can't find the container with id adea93393b2119572382b063d5ec5d82da0f3f1f6a3b33c582dc5fea4575fda1
Sep 06 10:21:50.383109 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.383090    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:21:50.434516 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.434476    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:50.446582 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.446556    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 10:21:50.447712 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.447694    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal"
Sep 06 10:21:50.453605 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.453591    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 10:21:50.691322 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:50.691237    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:51.111809 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.111728    2583 apiserver.go:52] "Watching apiserver"
Sep 06 10:21:51.117878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.117846    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 06 10:21:51.119096 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.119059    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl","openshift-cluster-node-tuning-operator/tuned-gfmdz","openshift-image-registry/node-ca-vd727","openshift-multus/multus-9c9ns","openshift-multus/multus-additional-cni-plugins-5lcn9","openshift-multus/network-metrics-daemon-xpk2b","openshift-network-diagnostics/network-check-target-2j5x7","kube-system/konnectivity-agent-q7fls","kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal","openshift-dns/node-resolver-jcbhz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal","openshift-network-operator/iptables-alerter-j42tp","openshift-ovn-kubernetes/ovnkube-node-fcsrt","kube-system/global-pull-secret-syncer-ckgmx"]
Sep 06 10:21:51.122024 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.122000    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:51.122149 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.122080    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:21:51.124280 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.124262    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.126777 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.126755    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fmthv\""
Sep 06 10:21:51.126969 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.126947    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 06 10:21:51.128243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.127171    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 06 10:21:51.128243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.127370    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 06 10:21:51.128243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.127921    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 06 10:21:51.128243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.127983    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 06 10:21:51.128243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.127927    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 06 10:21:51.130553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.129676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.134769 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.132183    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s5krt\""
Sep 06 10:21:51.134769 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.132418    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 06 10:21:51.134769 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.132600    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 06 10:21:51.134769 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.132780    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 06 10:21:51.134769 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.133032    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.134769 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.134503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.135283 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.134918    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tqvx9\""
Sep 06 10:21:51.135283 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.135018    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 06 10:21:51.135283 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.135233    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 06 10:21:51.135478 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.135328    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 06 10:21:51.135750 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.135563    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 06 10:21:51.135750 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.135631    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 06 10:21:51.136696 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.136678    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xtxfl\""
Sep 06 10:21:51.136932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.136692    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 06 10:21:51.136932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.136714    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 06 10:21:51.137083 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.137065    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:51.137174 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.137148    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:21:51.140273 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.140135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.140273 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.140230    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:21:51.146303 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-run-ovn-kubernetes\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.146474 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.146474 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf28k\" (UniqueName: \"kubernetes.io/projected/9b4c8743-d239-4ff4-97d3-55e1e861cc1e-kube-api-access-kf28k\") pod \"node-ca-vd727\" (UID: \"9b4c8743-d239-4ff4-97d3-55e1e861cc1e\") " pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.146474 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146454    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-var-lib-kubelet\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.146624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbxjk\" (UniqueName: \"kubernetes.io/projected/18abf90a-c64a-41a5-b6f9-836b95c43600-kube-api-access-hbxjk\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:51.146624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146518    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-run-netns\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.146624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146553    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-run-systemd\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.146624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146578    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2f9b6b1-a0fc-4069-9b0a-d67136690508-ovnkube-config\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.146624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146617    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2f9b6b1-a0fc-4069-9b0a-d67136690508-env-overrides\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.146850 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146648    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2f9b6b1-a0fc-4069-9b0a-d67136690508-ovn-node-metrics-cert\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.146850 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv7tp\" (UniqueName: \"kubernetes.io/projected/56811c57-413e-474d-9744-5827780e111d-kube-api-access-zv7tp\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.146850 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146712    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:51.146850 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-kubelet\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.146850 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-cni-dir\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.146850 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146800    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-socket-dir-parent\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.146850 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-run-k8s-cni-cncf-io\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-run-netns\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-conf-dir\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-etc-kubernetes\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b4c8743-d239-4ff4-97d3-55e1e861cc1e-host\") pod \"node-ca-vd727\" (UID: \"9b4c8743-d239-4ff4-97d3-55e1e861cc1e\") " pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.146983    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5xmt\" (UniqueName: \"kubernetes.io/projected/f4a955c6-baeb-4527-afc5-0bb8166c0f75-kube-api-access-q5xmt\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147012    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147040    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-var-lib-openvswitch\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-run-openvswitch\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147105    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-cni-bin\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.147157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-cni-netd\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.147682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147295    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2f9b6b1-a0fc-4069-9b0a-d67136690508-ovnkube-script-lib\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.147682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147361    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b4c8743-d239-4ff4-97d3-55e1e861cc1e-serviceca\") pod \"node-ca-vd727\" (UID: \"9b4c8743-d239-4ff4-97d3-55e1e861cc1e\") " pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.147682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147400    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.147682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147455    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-system-cni-dir\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.147682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-var-lib-cni-bin\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.147682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147575    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-slash\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.147682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-cnibin\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.147682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-hostroot\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.148038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-run-multus-certs\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.148038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56811c57-413e-474d-9744-5827780e111d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.148038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147817    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56811c57-413e-474d-9744-5827780e111d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.148038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56811c57-413e-474d-9744-5827780e111d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.148038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147948    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-systemd-units\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.148038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.147979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-etc-openvswitch\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.148038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4a955c6-baeb-4527-afc5-0bb8166c0f75-cni-binary-copy\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.148352 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-run-ovn\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.148352 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-node-log\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.148352 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnksn\" (UniqueName: \"kubernetes.io/projected/d2f9b6b1-a0fc-4069-9b0a-d67136690508-kube-api-access-lnksn\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.148352 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-var-lib-cni-multus\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.148352 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-daemon-config\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.148352 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148273    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56811c57-413e-474d-9744-5827780e111d-system-cni-dir\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.148622 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148384    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56811c57-413e-474d-9744-5827780e111d-cnibin\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.148622 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148424    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56811c57-413e-474d-9744-5827780e111d-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.148622 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.148457    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-log-socket\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.149362 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.149293    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.149915 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.149895    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2p28f\""
Sep 06 10:21:51.150010 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.149912    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 06 10:21:51.150097 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.150081    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 06 10:21:51.150161 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.150132    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 06 10:21:51.151401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.151380    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 06 10:21:51.152144 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.152120    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 06 10:21:51.152224 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.152157    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6s98h\""
Sep 06 10:21:51.155530 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.155505    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 10:21:51.155927 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.155910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.157948 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.157929    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 06 10:21:51.158039 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.158016    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kwpll\""
Sep 06 10:21:51.158286 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.158271    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 06 10:21:51.158588 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.158571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:21:51.160448 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.160430    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hlscz\""
Sep 06 10:21:51.160878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.160860    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 06 10:21:51.161129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.161109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.162421 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.162402    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 06 10:21:51.162827 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.162806    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 06 10:21:51.163118 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.163102    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jpqk9\""
Sep 06 10:21:51.163382 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.163366    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 06 10:21:51.163382 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.163378    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 06 10:21:51.171108 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.171084    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-trf7q"
Sep 06 10:21:51.180532 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.180508    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-trf7q"
Sep 06 10:21:51.235868 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.235837    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 06 10:21:51.240901 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.240844    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal" event={"ID":"61346eed96f8fec99ad05cd0344256fa","Type":"ContainerStarted","Data":"adea93393b2119572382b063d5ec5d82da0f3f1f6a3b33c582dc5fea4575fda1"}
Sep 06 10:21:51.242703 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.242674    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal" event={"ID":"caa56ac7ebc4ca9eb8489a6688c0f5a0","Type":"ContainerStarted","Data":"56821d7f648d7586256a65ffb8957e8cda12ff8bc1748fdd285603de0b4f2bea"}
Sep 06 10:21:51.248874 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.248845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-sys\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.249030 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.248890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15b6b3de-8170-431a-b270-81ebb6c5f7e9-tmp-dir\") pod \"node-resolver-jcbhz\" (UID: \"15b6b3de-8170-431a-b270-81ebb6c5f7e9\") " pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.249030 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.248922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-var-lib-cni-multus\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.249030 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.248943    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-daemon-config\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.249030 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.248969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56811c57-413e-474d-9744-5827780e111d-cnibin\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.249030 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.248993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56811c57-413e-474d-9744-5827780e111d-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.249276 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-log-socket\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.249276 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/56811c57-413e-474d-9744-5827780e111d-cnibin\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.249276 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-run-ovn-kubernetes\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.249276 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249190    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-var-lib-cni-multus\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.249276 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249216    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-log-socket\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.249276 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249272    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2f9b6b1-a0fc-4069-9b0a-d67136690508-ovn-node-metrics-cert\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.249603 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249314    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-run\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.249603 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249360    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbxjk\" (UniqueName: \"kubernetes.io/projected/18abf90a-c64a-41a5-b6f9-836b95c43600-kube-api-access-hbxjk\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:51.249603 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249393    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-run-netns\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.249603 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249402    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-run-ovn-kubernetes\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.249603 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-run-systemd\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.249603 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249438    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2f9b6b1-a0fc-4069-9b0a-d67136690508-ovnkube-config\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.249997 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.249955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-run-systemd\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.250051 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250010    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3445e67a-0762-4f21-b757-5621592527a8-iptables-alerter-script\") pod \"iptables-alerter-j42tp\" (UID: \"3445e67a-0762-4f21-b757-5621592527a8\") " pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.250051 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.250145 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250071    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-sysconfig\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.250145 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-lib-modules\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.250145 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250135    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv7tp\" (UniqueName: \"kubernetes.io/projected/56811c57-413e-474d-9744-5827780e111d-kube-api-access-zv7tp\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.250289 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-kubelet\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.250362 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250317    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-device-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.250423 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250389    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-host\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.250480 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/765079ac-964b-40e1-a47c-a7567108b78d-tmp\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.250534 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-cni-dir\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.250534 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-socket-dir-parent\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.250631 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250532    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/56811c57-413e-474d-9744-5827780e111d-cni-binary-copy\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.250631 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-run-k8s-cni-cncf-io\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.250631 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-etc-kubernetes\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.250631 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5xmt\" (UniqueName: \"kubernetes.io/projected/f4a955c6-baeb-4527-afc5-0bb8166c0f75-kube-api-access-q5xmt\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.250820 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250668    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-kubelet\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.250820 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250685    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:51.250820 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250737    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3445e67a-0762-4f21-b757-5621592527a8-host-slash\") pod \"iptables-alerter-j42tp\" (UID: \"3445e67a-0762-4f21-b757-5621592527a8\") " pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.250820 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b4c8743-d239-4ff4-97d3-55e1e861cc1e-host\") pod \"node-ca-vd727\" (UID: \"9b4c8743-d239-4ff4-97d3-55e1e861cc1e\") " pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.251002 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-var-lib-openvswitch\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.251002 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250900    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2f9b6b1-a0fc-4069-9b0a-d67136690508-ovnkube-script-lib\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.251002 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250961    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/431929b2-d956-4151-acea-b8d910e8c6fc-dbus\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.251144 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.250902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-daemon-config\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.251144 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-run-k8s-cni-cncf-io\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.251229 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251148    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b4c8743-d239-4ff4-97d3-55e1e861cc1e-host\") pod \"node-ca-vd727\" (UID: \"9b4c8743-d239-4ff4-97d3-55e1e861cc1e\") " pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.251279 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251259    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-cni-dir\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.251279 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-socket-dir-parent\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.251404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251305    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-kubernetes\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.251404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2f9b6b1-a0fc-4069-9b0a-d67136690508-ovnkube-config\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.251404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-etc-kubernetes\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.251404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251377    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcwh\" (UniqueName: \"kubernetes.io/projected/15b6b3de-8170-431a-b270-81ebb6c5f7e9-kube-api-access-2qcwh\") pod \"node-resolver-jcbhz\" (UID: \"15b6b3de-8170-431a-b270-81ebb6c5f7e9\") " pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.251404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251377    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-var-lib-openvswitch\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.251640 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b4c8743-d239-4ff4-97d3-55e1e861cc1e-serviceca\") pod \"node-ca-vd727\" (UID: \"9b4c8743-d239-4ff4-97d3-55e1e861cc1e\") " pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.251640 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251496    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-run-netns\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.251640 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251509    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.251640 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251548    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-run-multus-certs\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.251640 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251633    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-systemd-units\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.251864 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251667    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-node-log\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.251864 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtqs\" (UniqueName: \"kubernetes.io/projected/3445e67a-0762-4f21-b757-5621592527a8-kube-api-access-2xtqs\") pod \"iptables-alerter-j42tp\" (UID: \"3445e67a-0762-4f21-b757-5621592527a8\") " pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.251864 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251730    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-systemd\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.251864 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4a955c6-baeb-4527-afc5-0bb8166c0f75-cni-binary-copy\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.251864 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnksn\" (UniqueName: \"kubernetes.io/projected/d2f9b6b1-a0fc-4069-9b0a-d67136690508-kube-api-access-lnksn\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.252150 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b4c8743-d239-4ff4-97d3-55e1e861cc1e-serviceca\") pod \"node-ca-vd727\" (UID: \"9b4c8743-d239-4ff4-97d3-55e1e861cc1e\") " pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.252150 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.251898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-run-multus-certs\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.252150 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252002    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa7523f5-51a6-4d51-ad6e-afef3666c35d-konnectivity-ca\") pod \"konnectivity-agent-q7fls\" (UID: \"aa7523f5-51a6-4d51-ad6e-afef3666c35d\") " pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:21:51.252150 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252003    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-node-log\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.252150 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-systemd-units\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.252401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252212    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 06 10:21:51.252401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-sysctl-conf\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.252401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56811c57-413e-474d-9744-5827780e111d-system-cni-dir\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.252401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252382    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/56811c57-413e-474d-9744-5827780e111d-system-cni-dir\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.252401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.252642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2f9b6b1-a0fc-4069-9b0a-d67136690508-env-overrides\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.252642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.252642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4a955c6-baeb-4527-afc5-0bb8166c0f75-cni-binary-copy\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.252642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252511    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-socket-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.252642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252617    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2f9b6b1-a0fc-4069-9b0a-d67136690508-ovnkube-script-lib\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.252877 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf28k\" (UniqueName: \"kubernetes.io/projected/9b4c8743-d239-4ff4-97d3-55e1e861cc1e-kube-api-access-kf28k\") pod \"node-ca-vd727\" (UID: \"9b4c8743-d239-4ff4-97d3-55e1e861cc1e\") " pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.252877 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-var-lib-kubelet\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.252877 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:51.252877 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-var-lib-kubelet\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.252877 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.252837    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:51.253067 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.252887    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-run-netns\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.253067 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.253048    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs podName:18abf90a-c64a-41a5-b6f9-836b95c43600 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:51.753017919 +0000 UTC m=+3.025659422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs") pod "network-metrics-daemon-xpk2b" (UID: "18abf90a-c64a-41a5-b6f9-836b95c43600") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:51.253243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2f9b6b1-a0fc-4069-9b0a-d67136690508-env-overrides\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.253369 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-run-netns\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.253369 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253307    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-conf-dir\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.253557 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253380    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-cni-netd\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.253557 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-multus-conf-dir\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.253557 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-var-lib-kubelet\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.253557 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253474    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-cni-netd\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.253557 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/15b6b3de-8170-431a-b270-81ebb6c5f7e9-hosts-file\") pod \"node-resolver-jcbhz\" (UID: \"15b6b3de-8170-431a-b270-81ebb6c5f7e9\") " pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253562    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-run-openvswitch\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-cni-bin\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253606    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-sys-fs\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-sysctl-d\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253658    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-run-openvswitch\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253666    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-cni-bin\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253675    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/765079ac-964b-40e1-a47c-a7567108b78d-etc-tuned\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-system-cni-dir\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-var-lib-cni-bin\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253858    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-host-var-lib-cni-bin\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.253941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.253863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-slash\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.254546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254219    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.254546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254249    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7gn\" (UniqueName: \"kubernetes.io/projected/765079ac-964b-40e1-a47c-a7567108b78d-kube-api-access-qh7gn\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.254546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-cnibin\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.254546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-host-slash\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.254546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254356    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-cnibin\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.254546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254380    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-system-cni-dir\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.254546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-hostroot\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.254876 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254598    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56811c57-413e-474d-9744-5827780e111d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.254876 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254625    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56811c57-413e-474d-9744-5827780e111d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.254876 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56811c57-413e-474d-9744-5827780e111d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.254876 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254709    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-etc-openvswitch\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.254876 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254731    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-run-ovn\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.254876 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa7523f5-51a6-4d51-ad6e-afef3666c35d-agent-certs\") pod \"konnectivity-agent-q7fls\" (UID: \"aa7523f5-51a6-4d51-ad6e-afef3666c35d\") " pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:21:51.254876 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/431929b2-d956-4151-acea-b8d910e8c6fc-kubelet-config\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.255467 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254888    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-registration-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.255467 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254930    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gmrx\" (UniqueName: \"kubernetes.io/projected/fd4feb1a-178e-452c-8833-3ccbe3f1436d-kube-api-access-9gmrx\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.255467 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254954    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-modprobe-d\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.255467 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.254530    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4a955c6-baeb-4527-afc5-0bb8166c0f75-hostroot\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.255467 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.255032    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-etc-openvswitch\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.255467 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.255184    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2f9b6b1-a0fc-4069-9b0a-d67136690508-run-ovn\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.255722 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.255705    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/56811c57-413e-474d-9744-5827780e111d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.255968 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.255947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/56811c57-413e-474d-9744-5827780e111d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.256140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.256097    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/56811c57-413e-474d-9744-5827780e111d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.259116 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.259092    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2f9b6b1-a0fc-4069-9b0a-d67136690508-ovn-node-metrics-cert\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.265035 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.265011    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:21:51.265180 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.265041    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:21:51.265180 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.265057    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zvwqh for pod openshift-network-diagnostics/network-check-target-2j5x7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:51.265180 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.265124    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh podName:07743b98-da6f-486a-abff-020a64af3133 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:51.76510465 +0000 UTC m=+3.037746148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zvwqh" (UniqueName: "kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh") pod "network-check-target-2j5x7" (UID: "07743b98-da6f-486a-abff-020a64af3133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:51.269378 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.269348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnksn\" (UniqueName: \"kubernetes.io/projected/d2f9b6b1-a0fc-4069-9b0a-d67136690508-kube-api-access-lnksn\") pod \"ovnkube-node-fcsrt\" (UID: \"d2f9b6b1-a0fc-4069-9b0a-d67136690508\") " pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.270050 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.270028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv7tp\" (UniqueName: \"kubernetes.io/projected/56811c57-413e-474d-9744-5827780e111d-kube-api-access-zv7tp\") pod \"multus-additional-cni-plugins-5lcn9\" (UID: \"56811c57-413e-474d-9744-5827780e111d\") " pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.270502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.270478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5xmt\" (UniqueName: \"kubernetes.io/projected/f4a955c6-baeb-4527-afc5-0bb8166c0f75-kube-api-access-q5xmt\") pod \"multus-9c9ns\" (UID: \"f4a955c6-baeb-4527-afc5-0bb8166c0f75\") " pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.271502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.271480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbxjk\" (UniqueName: \"kubernetes.io/projected/18abf90a-c64a-41a5-b6f9-836b95c43600-kube-api-access-hbxjk\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:51.271651 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.271625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf28k\" (UniqueName: \"kubernetes.io/projected/9b4c8743-d239-4ff4-97d3-55e1e861cc1e-kube-api-access-kf28k\") pod \"node-ca-vd727\" (UID: \"9b4c8743-d239-4ff4-97d3-55e1e861cc1e\") " pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.355858 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.355810    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-socket-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.356041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.355880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-var-lib-kubelet\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.355906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/15b6b3de-8170-431a-b270-81ebb6c5f7e9-hosts-file\") pod \"node-resolver-jcbhz\" (UID: \"15b6b3de-8170-431a-b270-81ebb6c5f7e9\") " pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.356041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.355933    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-sys-fs\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.356041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.355970    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-sysctl-d\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.355993    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/765079ac-964b-40e1-a47c-a7567108b78d-etc-tuned\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356048    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7gn\" (UniqueName: \"kubernetes.io/projected/765079ac-964b-40e1-a47c-a7567108b78d-kube-api-access-qh7gn\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356079    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa7523f5-51a6-4d51-ad6e-afef3666c35d-agent-certs\") pod \"konnectivity-agent-q7fls\" (UID: \"aa7523f5-51a6-4d51-ad6e-afef3666c35d\") " pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/431929b2-d956-4151-acea-b8d910e8c6fc-kubelet-config\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-registration-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356133    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-socket-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gmrx\" (UniqueName: \"kubernetes.io/projected/fd4feb1a-178e-452c-8833-3ccbe3f1436d-kube-api-access-9gmrx\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356175    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-modprobe-d\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356196    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-sys\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15b6b3de-8170-431a-b270-81ebb6c5f7e9-tmp-dir\") pod \"node-resolver-jcbhz\" (UID: \"15b6b3de-8170-431a-b270-81ebb6c5f7e9\") " pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-run\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.356263    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3445e67a-0762-4f21-b757-5621592527a8-iptables-alerter-script\") pod \"iptables-alerter-j42tp\" (UID: \"3445e67a-0762-4f21-b757-5621592527a8\") " pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356297    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.356321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356311    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-sysconfig\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.356358    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret podName:431929b2-d956-4151-acea-b8d910e8c6fc nodeName:}" failed. No retries permitted until 2026-09-06 10:21:51.856316494 +0000 UTC m=+3.128957988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret") pod "global-pull-secret-syncer-ckgmx" (UID: "431929b2-d956-4151-acea-b8d910e8c6fc") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-lib-modules\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356427    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-device-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-host\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/765079ac-964b-40e1-a47c-a7567108b78d-tmp\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356537    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3445e67a-0762-4f21-b757-5621592527a8-host-slash\") pod \"iptables-alerter-j42tp\" (UID: \"3445e67a-0762-4f21-b757-5621592527a8\") " pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/431929b2-d956-4151-acea-b8d910e8c6fc-dbus\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-kubernetes\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356679    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcwh\" (UniqueName: \"kubernetes.io/projected/15b6b3de-8170-431a-b270-81ebb6c5f7e9-kube-api-access-2qcwh\") pod \"node-resolver-jcbhz\" (UID: \"15b6b3de-8170-431a-b270-81ebb6c5f7e9\") " pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356693    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-sys\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356739    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtqs\" (UniqueName: \"kubernetes.io/projected/3445e67a-0762-4f21-b757-5621592527a8-kube-api-access-2xtqs\") pod \"iptables-alerter-j42tp\" (UID: \"3445e67a-0762-4f21-b757-5621592527a8\") " pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-modprobe-d\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-systemd\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa7523f5-51a6-4d51-ad6e-afef3666c35d-konnectivity-ca\") pod \"konnectivity-agent-q7fls\" (UID: \"aa7523f5-51a6-4d51-ad6e-afef3666c35d\") " pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-sysctl-conf\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.356989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356820    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-systemd\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357000    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15b6b3de-8170-431a-b270-81ebb6c5f7e9-tmp-dir\") pod \"node-resolver-jcbhz\" (UID: \"15b6b3de-8170-431a-b270-81ebb6c5f7e9\") " pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-sysctl-conf\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-run\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-var-lib-kubelet\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357236    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/15b6b3de-8170-431a-b270-81ebb6c5f7e9-hosts-file\") pod \"node-resolver-jcbhz\" (UID: \"15b6b3de-8170-431a-b270-81ebb6c5f7e9\") " pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-sys-fs\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-etc-selinux\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357358    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-registration-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357418    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/431929b2-d956-4151-acea-b8d910e8c6fc-kubelet-config\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357412    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-sysctl-d\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357582    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3445e67a-0762-4f21-b757-5621592527a8-iptables-alerter-script\") pod \"iptables-alerter-j42tp\" (UID: \"3445e67a-0762-4f21-b757-5621592527a8\") " pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.356395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-sysconfig\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-etc-kubernetes\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357786    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-device-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-host\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.357797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357809    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd4feb1a-178e-452c-8833-3ccbe3f1436d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.358553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357834    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3445e67a-0762-4f21-b757-5621592527a8-host-slash\") pod \"iptables-alerter-j42tp\" (UID: \"3445e67a-0762-4f21-b757-5621592527a8\") " pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.358553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/765079ac-964b-40e1-a47c-a7567108b78d-lib-modules\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.358553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.357930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/431929b2-d956-4151-acea-b8d910e8c6fc-dbus\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.358553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.358152    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa7523f5-51a6-4d51-ad6e-afef3666c35d-konnectivity-ca\") pod \"konnectivity-agent-q7fls\" (UID: \"aa7523f5-51a6-4d51-ad6e-afef3666c35d\") " pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:21:51.360100 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.360076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/765079ac-964b-40e1-a47c-a7567108b78d-tmp\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.360299 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.360282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa7523f5-51a6-4d51-ad6e-afef3666c35d-agent-certs\") pod \"konnectivity-agent-q7fls\" (UID: \"aa7523f5-51a6-4d51-ad6e-afef3666c35d\") " pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:21:51.360577 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.360553    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/765079ac-964b-40e1-a47c-a7567108b78d-etc-tuned\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.369721 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.369654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gmrx\" (UniqueName: \"kubernetes.io/projected/fd4feb1a-178e-452c-8833-3ccbe3f1436d-kube-api-access-9gmrx\") pod \"aws-ebs-csi-driver-node-cqbkl\" (UID: \"fd4feb1a-178e-452c-8833-3ccbe3f1436d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.369865 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.369717    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtqs\" (UniqueName: \"kubernetes.io/projected/3445e67a-0762-4f21-b757-5621592527a8-kube-api-access-2xtqs\") pod \"iptables-alerter-j42tp\" (UID: \"3445e67a-0762-4f21-b757-5621592527a8\") " pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.369933 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.369887    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7gn\" (UniqueName: \"kubernetes.io/projected/765079ac-964b-40e1-a47c-a7567108b78d-kube-api-access-qh7gn\") pod \"tuned-gfmdz\" (UID: \"765079ac-964b-40e1-a47c-a7567108b78d\") " pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.370017 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.369993    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcwh\" (UniqueName: \"kubernetes.io/projected/15b6b3de-8170-431a-b270-81ebb6c5f7e9-kube-api-access-2qcwh\") pod \"node-resolver-jcbhz\" (UID: \"15b6b3de-8170-431a-b270-81ebb6c5f7e9\") " pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.382959 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.382933    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:51.440313 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.440271    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:21:51.456241 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.456211    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vd727"
Sep 06 10:21:51.466163 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.466135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9c9ns"
Sep 06 10:21:51.471996 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.471971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5lcn9"
Sep 06 10:21:51.479941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.479915    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl"
Sep 06 10:21:51.494763 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.494734    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gfmdz"
Sep 06 10:21:51.503648 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.503620    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jcbhz"
Sep 06 10:21:51.510489 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.510462    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:21:51.518217 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.518191    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-j42tp"
Sep 06 10:21:51.759681 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.759587    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:51.759822 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.759763    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:51.759878 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.759852    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs podName:18abf90a-c64a-41a5-b6f9-836b95c43600 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:52.759824362 +0000 UTC m=+4.032465846 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs") pod "network-metrics-daemon-xpk2b" (UID: "18abf90a-c64a-41a5-b6f9-836b95c43600") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:51.860100 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.860067    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:51.860300 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.860208    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:51.860300 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:51.860232    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:51.860300 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.860290    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret podName:431929b2-d956-4151-acea-b8d910e8c6fc nodeName:}" failed. No retries permitted until 2026-09-06 10:21:52.860269812 +0000 UTC m=+4.132911316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret") pod "global-pull-secret-syncer-ckgmx" (UID: "431929b2-d956-4151-acea-b8d910e8c6fc") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:51.860492 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.860434    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:21:51.860492 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.860457    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:21:51.860492 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.860481    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zvwqh for pod openshift-network-diagnostics/network-check-target-2j5x7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:51.860643 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:51.860532    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh podName:07743b98-da6f-486a-abff-020a64af3133 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:52.860515868 +0000 UTC m=+4.133157372 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zvwqh" (UniqueName: "kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh") pod "network-check-target-2j5x7" (UID: "07743b98-da6f-486a-abff-020a64af3133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:52.167357 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:52.167259    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:21:52.183317 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:52.183280    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 10:16:51 +0000 UTC" deadline="2028-05-11 19:13:11.469560077 +0000 UTC"
Sep 06 10:21:52.183317 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:52.183312    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14720h51m19.286252084s"
Sep 06 10:21:52.412988 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:52.412960    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7523f5_51a6_4d51_ad6e_afef3666c35d.slice/crio-52b0fa244fb897d767f0034bd10c2aca0f925e019c80e10c906a3be562e77227 WatchSource:0}: Error finding container 52b0fa244fb897d767f0034bd10c2aca0f925e019c80e10c906a3be562e77227: Status 404 returned error can't find the container with id 52b0fa244fb897d767f0034bd10c2aca0f925e019c80e10c906a3be562e77227
Sep 06 10:21:52.414906 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:52.414879    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765079ac_964b_40e1_a47c_a7567108b78d.slice/crio-c9e1222eadf56fb3564b9949c781262c226cecc43d273b96d1038c116f591055 WatchSource:0}: Error finding container c9e1222eadf56fb3564b9949c781262c226cecc43d273b96d1038c116f591055: Status 404 returned error can't find the container with id c9e1222eadf56fb3564b9949c781262c226cecc43d273b96d1038c116f591055
Sep 06 10:21:52.415494 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:52.415161    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f9b6b1_a0fc_4069_9b0a_d67136690508.slice/crio-36e8d3d1ba8b2982012307fa14bd306cd100b15021156db0b742725caa22172d WatchSource:0}: Error finding container 36e8d3d1ba8b2982012307fa14bd306cd100b15021156db0b742725caa22172d: Status 404 returned error can't find the container with id 36e8d3d1ba8b2982012307fa14bd306cd100b15021156db0b742725caa22172d
Sep 06 10:21:52.417005 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:52.416988    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3445e67a_0762_4f21_b757_5621592527a8.slice/crio-39bbec68c569eb831a86b2b4f8495e6f10c8b119e429a9210e842c1e933f29fc WatchSource:0}: Error finding container 39bbec68c569eb831a86b2b4f8495e6f10c8b119e429a9210e842c1e933f29fc: Status 404 returned error can't find the container with id 39bbec68c569eb831a86b2b4f8495e6f10c8b119e429a9210e842c1e933f29fc
Sep 06 10:21:52.420083 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:52.420060    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a955c6_baeb_4527_afc5_0bb8166c0f75.slice/crio-48a8de3a1f429a5a7ccba747108a6dedce449d84608336c91087a2f832e32b9b WatchSource:0}: Error finding container 48a8de3a1f429a5a7ccba747108a6dedce449d84608336c91087a2f832e32b9b: Status 404 returned error can't find the container with id 48a8de3a1f429a5a7ccba747108a6dedce449d84608336c91087a2f832e32b9b
Sep 06 10:21:52.421230 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:52.421189    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b6b3de_8170_431a_b270_81ebb6c5f7e9.slice/crio-2bd534cb48d49e1b54934b364be157a65366ea5e2636dfdf00590fbdc5347c29 WatchSource:0}: Error finding container 2bd534cb48d49e1b54934b364be157a65366ea5e2636dfdf00590fbdc5347c29: Status 404 returned error can't find the container with id 2bd534cb48d49e1b54934b364be157a65366ea5e2636dfdf00590fbdc5347c29
Sep 06 10:21:52.422908 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:52.422884    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4feb1a_178e_452c_8833_3ccbe3f1436d.slice/crio-5ec5fbc28011ccec4b5c4caf2d6368d8a9272b5c98b60835a968e21b8dbf3df2 WatchSource:0}: Error finding container 5ec5fbc28011ccec4b5c4caf2d6368d8a9272b5c98b60835a968e21b8dbf3df2: Status 404 returned error can't find the container with id 5ec5fbc28011ccec4b5c4caf2d6368d8a9272b5c98b60835a968e21b8dbf3df2
Sep 06 10:21:52.425281 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:52.424588    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56811c57_413e_474d_9744_5827780e111d.slice/crio-c68608a92ee7abb5880e4518ffe6751723d43c05753a57f49a719d183c820b8f WatchSource:0}: Error finding container c68608a92ee7abb5880e4518ffe6751723d43c05753a57f49a719d183c820b8f: Status 404 returned error can't find the container with id c68608a92ee7abb5880e4518ffe6751723d43c05753a57f49a719d183c820b8f
Sep 06 10:21:52.426959 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:21:52.426927    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4c8743_d239_4ff4_97d3_55e1e861cc1e.slice/crio-8dac833f2eebe86238e50afca3320e79796bb61743d920875156ce9df2870d24 WatchSource:0}: Error finding container 8dac833f2eebe86238e50afca3320e79796bb61743d920875156ce9df2870d24: Status 404 returned error can't find the container with id 8dac833f2eebe86238e50afca3320e79796bb61743d920875156ce9df2870d24
Sep 06 10:21:52.767498 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:52.767428    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:52.767669 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:52.767599    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:52.767735 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:52.767686    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs podName:18abf90a-c64a-41a5-b6f9-836b95c43600 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:54.767665007 +0000 UTC m=+6.040306491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs") pod "network-metrics-daemon-xpk2b" (UID: "18abf90a-c64a-41a5-b6f9-836b95c43600") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:52.869303 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:52.868729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:52.869303 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:52.868814    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:52.869303 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:52.868940    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:52.869303 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:52.869002    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret podName:431929b2-d956-4151-acea-b8d910e8c6fc nodeName:}" failed. No retries permitted until 2026-09-06 10:21:54.868984784 +0000 UTC m=+6.141626287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret") pod "global-pull-secret-syncer-ckgmx" (UID: "431929b2-d956-4151-acea-b8d910e8c6fc") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:52.869303 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:52.869091    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:21:52.869303 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:52.869115    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:21:52.869303 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:52.869129    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zvwqh for pod openshift-network-diagnostics/network-check-target-2j5x7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:52.869303 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:52.869211    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh podName:07743b98-da6f-486a-abff-020a64af3133 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:54.869165362 +0000 UTC m=+6.141806843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zvwqh" (UniqueName: "kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh") pod "network-check-target-2j5x7" (UID: "07743b98-da6f-486a-abff-020a64af3133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:53.183581 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.183503    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 10:16:51 +0000 UTC" deadline="2028-06-12 16:31:41.038161488 +0000 UTC"
Sep 06 10:21:53.183581 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.183571    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15486h9m47.854594196s"
Sep 06 10:21:53.235502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.235465    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:53.235696 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:53.235621    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:21:53.236047 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.236026    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:53.236144 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:53.236122    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:21:53.237907 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.237885    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:53.238039 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:53.237983    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:21:53.262237 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.262197    2583 generic.go:356] "Generic (PLEG): container finished" podID="61346eed96f8fec99ad05cd0344256fa" containerID="793738c0a3315ac6dd6adf0ea3286c936159373787020c89384171b9660222a8" exitCode=0
Sep 06 10:21:53.262441 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.262306    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal" event={"ID":"61346eed96f8fec99ad05cd0344256fa","Type":"ContainerDied","Data":"793738c0a3315ac6dd6adf0ea3286c936159373787020c89384171b9660222a8"}
Sep 06 10:21:53.292191 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.292149    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal" event={"ID":"caa56ac7ebc4ca9eb8489a6688c0f5a0","Type":"ContainerStarted","Data":"9b68f19c92cff6a1fefa182e12046fb3e608a864fbaea94a4df0f066b338a0d6"}
Sep 06 10:21:53.299676 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.299616    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jcbhz" event={"ID":"15b6b3de-8170-431a-b270-81ebb6c5f7e9","Type":"ContainerStarted","Data":"2bd534cb48d49e1b54934b364be157a65366ea5e2636dfdf00590fbdc5347c29"}
Sep 06 10:21:53.306106 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.306000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lcn9" event={"ID":"56811c57-413e-474d-9744-5827780e111d","Type":"ContainerStarted","Data":"c68608a92ee7abb5880e4518ffe6751723d43c05753a57f49a719d183c820b8f"}
Sep 06 10:21:53.316826 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.316789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vd727" event={"ID":"9b4c8743-d239-4ff4-97d3-55e1e861cc1e","Type":"ContainerStarted","Data":"8dac833f2eebe86238e50afca3320e79796bb61743d920875156ce9df2870d24"}
Sep 06 10:21:53.333311 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.333254    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9c9ns" event={"ID":"f4a955c6-baeb-4527-afc5-0bb8166c0f75","Type":"ContainerStarted","Data":"48a8de3a1f429a5a7ccba747108a6dedce449d84608336c91087a2f832e32b9b"}
Sep 06 10:21:53.348821 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.348776    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" event={"ID":"d2f9b6b1-a0fc-4069-9b0a-d67136690508","Type":"ContainerStarted","Data":"36e8d3d1ba8b2982012307fa14bd306cd100b15021156db0b742725caa22172d"}
Sep 06 10:21:53.363245 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.363115    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7fls" event={"ID":"aa7523f5-51a6-4d51-ad6e-afef3666c35d","Type":"ContainerStarted","Data":"52b0fa244fb897d767f0034bd10c2aca0f925e019c80e10c906a3be562e77227"}
Sep 06 10:21:53.373291 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.373186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl" event={"ID":"fd4feb1a-178e-452c-8833-3ccbe3f1436d","Type":"ContainerStarted","Data":"5ec5fbc28011ccec4b5c4caf2d6368d8a9272b5c98b60835a968e21b8dbf3df2"}
Sep 06 10:21:53.386826 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.386787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j42tp" event={"ID":"3445e67a-0762-4f21-b757-5621592527a8","Type":"ContainerStarted","Data":"39bbec68c569eb831a86b2b4f8495e6f10c8b119e429a9210e842c1e933f29fc"}
Sep 06 10:21:53.398068 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:53.398017    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gfmdz" event={"ID":"765079ac-964b-40e1-a47c-a7567108b78d","Type":"ContainerStarted","Data":"c9e1222eadf56fb3564b9949c781262c226cecc43d273b96d1038c116f591055"}
Sep 06 10:21:54.422534 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:54.422112    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal" event={"ID":"61346eed96f8fec99ad05cd0344256fa","Type":"ContainerStarted","Data":"b46a120aaec373a72d264f645f711bada5bc9c5c83890b801bdb438484c06d63"}
Sep 06 10:21:54.438203 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:54.438139    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-109.ec2.internal" podStartSLOduration=4.438117024 podStartE2EDuration="4.438117024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:21:54.438000366 +0000 UTC m=+5.710641870" watchObservedRunningTime="2026-09-06 10:21:54.438117024 +0000 UTC m=+5.710758526"
Sep 06 10:21:54.439196 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:54.439145    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-109.ec2.internal" podStartSLOduration=4.43912863 podStartE2EDuration="4.43912863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:21:53.308222921 +0000 UTC m=+4.580864425" watchObservedRunningTime="2026-09-06 10:21:54.43912863 +0000 UTC m=+5.711770134"
Sep 06 10:21:54.786310 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:54.785648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:54.786310 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:54.785833    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:54.786310 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:54.785899    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs podName:18abf90a-c64a-41a5-b6f9-836b95c43600 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:58.785881882 +0000 UTC m=+10.058523367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs") pod "network-metrics-daemon-xpk2b" (UID: "18abf90a-c64a-41a5-b6f9-836b95c43600") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:54.887202 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:54.886417    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:54.887202 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:54.886497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:54.887202 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:54.886632    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:54.887202 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:54.886695    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret podName:431929b2-d956-4151-acea-b8d910e8c6fc nodeName:}" failed. No retries permitted until 2026-09-06 10:21:58.886677454 +0000 UTC m=+10.159318941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret") pod "global-pull-secret-syncer-ckgmx" (UID: "431929b2-d956-4151-acea-b8d910e8c6fc") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:54.887202 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:54.887087    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:21:54.887202 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:54.887107    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:21:54.887202 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:54.887119    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zvwqh for pod openshift-network-diagnostics/network-check-target-2j5x7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:54.887202 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:54.887163    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh podName:07743b98-da6f-486a-abff-020a64af3133 nodeName:}" failed. No retries permitted until 2026-09-06 10:21:58.887148766 +0000 UTC m=+10.159790261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zvwqh" (UniqueName: "kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh") pod "network-check-target-2j5x7" (UID: "07743b98-da6f-486a-abff-020a64af3133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:55.235214 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:55.235135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:55.235381 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:55.235291    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:21:55.235705 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:55.235682    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:55.235833 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:55.235810    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:21:55.236229 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:55.236206    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:55.236328 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:55.236306    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:21:57.234761 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:57.234720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:57.235236 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:57.234898    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:21:57.235236 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:57.235003    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:57.235236 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:57.235111    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:21:57.235450 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:57.235269    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:57.235450 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:57.235379    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:21:58.820459 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:58.820413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:58.820965 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:58.820586    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:58.820965 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:58.820661    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs podName:18abf90a-c64a-41a5-b6f9-836b95c43600 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:06.820642964 +0000 UTC m=+18.093284451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs") pod "network-metrics-daemon-xpk2b" (UID: "18abf90a-c64a-41a5-b6f9-836b95c43600") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:21:58.920931 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:58.920888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:58.921115 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:58.920957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:58.921193 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:58.921124    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:21:58.921193 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:58.921150    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:21:58.921193 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:58.921161    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zvwqh for pod openshift-network-diagnostics/network-check-target-2j5x7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:58.921424 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:58.921207    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh podName:07743b98-da6f-486a-abff-020a64af3133 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:06.921191563 +0000 UTC m=+18.193833043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zvwqh" (UniqueName: "kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh") pod "network-check-target-2j5x7" (UID: "07743b98-da6f-486a-abff-020a64af3133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:21:58.921498 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:58.921426    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:58.921498 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:58.921464    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret podName:431929b2-d956-4151-acea-b8d910e8c6fc nodeName:}" failed. No retries permitted until 2026-09-06 10:22:06.921453246 +0000 UTC m=+18.194094727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret") pod "global-pull-secret-syncer-ckgmx" (UID: "431929b2-d956-4151-acea-b8d910e8c6fc") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:21:59.235707 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:59.235621    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:21:59.235982 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:59.235707    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:21:59.236125 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:59.236100    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:21:59.236191 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:21:59.235740    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:21:59.236242 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:59.236212    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:21:59.236323 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:21:59.236117    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:01.234487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:01.234444    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:01.234934 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:01.234583    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:01.234934 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:01.234676    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:01.234934 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:01.234784    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:01.234934 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:01.234845    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:01.235083 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:01.234935    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:03.234541 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:03.234503    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:03.234541 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:03.234529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:03.235007 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:03.234506    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:03.235007 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:03.234694    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:03.235007 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:03.234764    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:03.235007 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:03.234838    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:05.233992 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:05.233952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:05.234471 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:05.233952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:05.234471 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:05.234078    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:05.234471 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:05.234209    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:05.234471 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:05.233953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:05.234471 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:05.234352    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:06.882372 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:06.882321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:06.882748 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:06.882463    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:06.882748 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:06.882524    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs podName:18abf90a-c64a-41a5-b6f9-836b95c43600 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:22.882507916 +0000 UTC m=+34.155149410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs") pod "network-metrics-daemon-xpk2b" (UID: "18abf90a-c64a-41a5-b6f9-836b95c43600") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:06.983279 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:06.983235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:06.983488 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:06.983421    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:06.983488 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:06.983420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:06.983600 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:06.983497    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret podName:431929b2-d956-4151-acea-b8d910e8c6fc nodeName:}" failed. No retries permitted until 2026-09-06 10:22:22.983476966 +0000 UTC m=+34.256118447 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret") pod "global-pull-secret-syncer-ckgmx" (UID: "431929b2-d956-4151-acea-b8d910e8c6fc") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:06.983600 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:06.983524    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:22:06.983600 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:06.983547    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:22:06.983600 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:06.983560    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zvwqh for pod openshift-network-diagnostics/network-check-target-2j5x7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:06.983788 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:06.983614    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh podName:07743b98-da6f-486a-abff-020a64af3133 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:22.983599322 +0000 UTC m=+34.256240806 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zvwqh" (UniqueName: "kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh") pod "network-check-target-2j5x7" (UID: "07743b98-da6f-486a-abff-020a64af3133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:07.234197 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:07.234097    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:07.234197 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:07.234140    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:07.234197 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:07.234109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:07.234467 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:07.234217    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:07.234467 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:07.234304    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:07.234467 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:07.234399    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:09.234935 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:09.234897    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:09.235422 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:09.235000    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:09.235422 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:09.235012    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:09.235422 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:09.235113    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:09.235422 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:09.235178    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:09.235422 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:09.235290    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:11.234826 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.234574    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:11.235682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.234586    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:11.235682 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:11.234919    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:11.235682 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:11.235015    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:11.235682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.234647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:11.235682 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:11.235153    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:11.470899 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.470663    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" event={"ID":"d2f9b6b1-a0fc-4069-9b0a-d67136690508","Type":"ContainerStarted","Data":"2ac2a54b1f25dca97416b5d01ed3e505acc4312aaf562d22e641abed7bb0a798"}
Sep 06 10:22:11.471062 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.470964    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" event={"ID":"d2f9b6b1-a0fc-4069-9b0a-d67136690508","Type":"ContainerStarted","Data":"06e74bad679784bc5b31900bf3de1f460511a172f467e2de8128b04ccc198f54"}
Sep 06 10:22:11.471062 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.471020    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" event={"ID":"d2f9b6b1-a0fc-4069-9b0a-d67136690508","Type":"ContainerStarted","Data":"3477cd86dbb15a6a88ab7eaa1303d7a8ce3b0c1a6ebac050171bd306f367fbe0"}
Sep 06 10:22:11.471208 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.471067    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" event={"ID":"d2f9b6b1-a0fc-4069-9b0a-d67136690508","Type":"ContainerStarted","Data":"a34b89495af89630b5fcfac514e59499fcb4df6beb687ef76773f34768a6cdee"}
Sep 06 10:22:11.471208 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.471110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" event={"ID":"d2f9b6b1-a0fc-4069-9b0a-d67136690508","Type":"ContainerStarted","Data":"6dde93ce62611f9ef21d6dad6d171ec92e947407f09737c56ebfc0f74f439018"}
Sep 06 10:22:11.471208 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.471151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" event={"ID":"d2f9b6b1-a0fc-4069-9b0a-d67136690508","Type":"ContainerStarted","Data":"b505ff4fcfd1854cf32b41574af3665890985bf0660f0f8f0a597796e02e83b8"}
Sep 06 10:22:11.472437 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.472392    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q7fls" event={"ID":"aa7523f5-51a6-4d51-ad6e-afef3666c35d","Type":"ContainerStarted","Data":"ee3a0a083d967c5314f603b71316772a94fa714dc9a9f5d7026a8a9222df025e"}
Sep 06 10:22:11.474363 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.474319    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl" event={"ID":"fd4feb1a-178e-452c-8833-3ccbe3f1436d","Type":"ContainerStarted","Data":"7f0c05837352fca3739fd30043e8b71d36239991217ed47f64797f106f597c45"}
Sep 06 10:22:11.476128 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.476102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gfmdz" event={"ID":"765079ac-964b-40e1-a47c-a7567108b78d","Type":"ContainerStarted","Data":"1b21292cd4ce395893c371697d9514d523a6eeb6a1bedfe912783d867032e2e6"}
Sep 06 10:22:11.478019 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.477994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jcbhz" event={"ID":"15b6b3de-8170-431a-b270-81ebb6c5f7e9","Type":"ContainerStarted","Data":"6e41fdf3b80603052d54e7fb5e16a5dadbe1f8bfc0e3a3b7d18a6ff53d72b082"}
Sep 06 10:22:11.479820 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.479793    2583 generic.go:356] "Generic (PLEG): container finished" podID="56811c57-413e-474d-9744-5827780e111d" containerID="474062b42ce479a46d2ea1aed6564fdae99535edf163a184628b89cf825c0ee6" exitCode=0
Sep 06 10:22:11.479924 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.479875    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lcn9" event={"ID":"56811c57-413e-474d-9744-5827780e111d","Type":"ContainerDied","Data":"474062b42ce479a46d2ea1aed6564fdae99535edf163a184628b89cf825c0ee6"}
Sep 06 10:22:11.481577 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.481557    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vd727" event={"ID":"9b4c8743-d239-4ff4-97d3-55e1e861cc1e","Type":"ContainerStarted","Data":"80d538652cf74df9c76b31951f43d62ec49102c97f8f357eab69a422cd26362d"}
Sep 06 10:22:11.483012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.482995    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9c9ns" event={"ID":"f4a955c6-baeb-4527-afc5-0bb8166c0f75","Type":"ContainerStarted","Data":"38855b8b604cd40e1143a46c33ad3bd5ac716babf9f25e2e72db321a9e8ebd6b"}
Sep 06 10:22:11.487567 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.487481    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q7fls" podStartSLOduration=8.054632027 podStartE2EDuration="22.487469065s" totalImagesPullingTime="14.432837038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:11.487208585 +0000 UTC m=+22.759850087" watchObservedRunningTime="2026-09-06 10:22:11.487469065 +0000 UTC m=+22.760110567"
Sep 06 10:22:11.534437 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.534386    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jcbhz" podStartSLOduration=4.640050645 podStartE2EDuration="22.534369269s" totalImagesPullingTime="17.894318624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:11.534359322 +0000 UTC m=+22.807000826" watchObservedRunningTime="2026-09-06 10:22:11.534369269 +0000 UTC m=+22.807010772"
Sep 06 10:22:11.534714 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.534694    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gfmdz" podStartSLOduration=4.611380975 podStartE2EDuration="22.534687867s" totalImagesPullingTime="17.923306892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:11.520681226 +0000 UTC m=+22.793322728" watchObservedRunningTime="2026-09-06 10:22:11.534687867 +0000 UTC m=+22.807329369"
Sep 06 10:22:11.553882 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.553835    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vd727" podStartSLOduration=4.663456501 podStartE2EDuration="22.553819364s" totalImagesPullingTime="17.890362863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:11.55346814 +0000 UTC m=+22.826109642" watchObservedRunningTime="2026-09-06 10:22:11.553819364 +0000 UTC m=+22.826460868"
Sep 06 10:22:11.569672 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.569622    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9c9ns" podStartSLOduration=4.619673953 podStartE2EDuration="22.569606684s" totalImagesPullingTime="17.949932731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:11.569029034 +0000 UTC m=+22.841670537" watchObservedRunningTime="2026-09-06 10:22:11.569606684 +0000 UTC m=+22.842248187"
Sep 06 10:22:11.853594 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:11.853565    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 06 10:22:12.215695 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:12.215522    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T10:22:11.8535846Z","UUID":"24a5b83f-b1e9-45d2-ab6f-1813998f004f","Handler":null,"Name":"","Endpoint":""}
Sep 06 10:22:12.217660 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:12.217637    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 06 10:22:12.217812 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:12.217671    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 06 10:22:12.487190 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:12.487105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-j42tp" event={"ID":"3445e67a-0762-4f21-b757-5621592527a8","Type":"ContainerStarted","Data":"8adb1dc1b49514663bb32aa46605936d0951a454d4511af7bf5737d57d5b9828"}
Sep 06 10:22:12.489461 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:12.489412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl" event={"ID":"fd4feb1a-178e-452c-8833-3ccbe3f1436d","Type":"ContainerStarted","Data":"84b4af99f4b5e7141d2b555a533c78a9ae5052ce117616b87114bdaa3e845ef0"}
Sep 06 10:22:12.501505 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:12.501447    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-j42tp" podStartSLOduration=5.599939232 podStartE2EDuration="23.501427896s" totalImagesPullingTime="17.901488664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:12.501208348 +0000 UTC m=+23.773849851" watchObservedRunningTime="2026-09-06 10:22:12.501427896 +0000 UTC m=+23.774069401"
Sep 06 10:22:13.234936 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:13.234847    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:13.235082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:13.234847    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:13.235082 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:13.234969    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:13.235082 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:13.235065    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:13.235217 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:13.234848    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:13.235217 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:13.235159    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:13.496249 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:13.496167    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" event={"ID":"d2f9b6b1-a0fc-4069-9b0a-d67136690508","Type":"ContainerStarted","Data":"fa5c7419ae6101ad4d48743d85a2bcfa8d9a1b3ed5c9b8769874f19a54ac176c"}
Sep 06 10:22:13.499142 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:13.499105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl" event={"ID":"fd4feb1a-178e-452c-8833-3ccbe3f1436d","Type":"ContainerStarted","Data":"d3c6fb3e9d930ece6fe4ee4014bfcd7a4118ae53239d3a6d150a0aa1ecb0ec6a"}
Sep 06 10:22:13.518272 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:13.518204    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cqbkl" podStartSLOduration=4.3438002860000005 podStartE2EDuration="24.518161484s" totalImagesPullingTime="20.174361198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:13.517503946 +0000 UTC m=+24.790145450" watchObservedRunningTime="2026-09-06 10:22:13.518161484 +0000 UTC m=+24.790802986"
Sep 06 10:22:13.754624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:13.754530    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:22:13.755852 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:13.755829    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:22:14.501930 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:14.501896    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:22:14.502449 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:14.502360    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q7fls"
Sep 06 10:22:15.234391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:15.234319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:15.234391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:15.234326    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:15.234629 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:15.234319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:15.234629 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:15.234463    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:15.234629 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:15.234538    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:15.234629 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:15.234602    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:16.507240 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:16.507049    2583 generic.go:356] "Generic (PLEG): container finished" podID="56811c57-413e-474d-9744-5827780e111d" containerID="e678d579fb760ef8b37baa200d1d6eb482851e2d6c6114d155a467fdc6454dcc" exitCode=0
Sep 06 10:22:16.508046 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:16.507140    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lcn9" event={"ID":"56811c57-413e-474d-9744-5827780e111d","Type":"ContainerDied","Data":"e678d579fb760ef8b37baa200d1d6eb482851e2d6c6114d155a467fdc6454dcc"}
Sep 06 10:22:16.513177 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:16.513154    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" event={"ID":"d2f9b6b1-a0fc-4069-9b0a-d67136690508","Type":"ContainerStarted","Data":"125e7ecf4bbfa0b6b259bd93a1370e230d1627483bb459949edb963287d41187"}
Sep 06 10:22:16.552318 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:16.552255    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt" podStartSLOduration=9.589351503 podStartE2EDuration="27.552237976s" totalImagesPullingTime="17.962886473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:16.552060779 +0000 UTC m=+27.824702282" watchObservedRunningTime="2026-09-06 10:22:16.552237976 +0000 UTC m=+27.824879479"
Sep 06 10:22:17.236561 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:17.236278    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:17.236773 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:17.236401    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:17.236773 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:17.236488    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:17.236773 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:17.236678    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:17.236773 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:17.236766    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:17.236991 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:17.236869    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:17.516060 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:17.515981    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:22:17.516511 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:17.516062    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:22:17.516511 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:17.516111    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:22:17.532916 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:17.532888    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:22:17.533078 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:17.533006    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:22:18.157394 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:18.157367    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ckgmx"]
Sep 06 10:22:18.157569 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:18.157473    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:18.157624 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:18.157579    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:18.160719 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:18.160692    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xpk2b"]
Sep 06 10:22:18.160935 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:18.160810    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:18.161009 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:18.160929    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:18.161523 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:18.161477    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2j5x7"]
Sep 06 10:22:18.161609 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:18.161556    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:18.161650 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:18.161623    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:18.520522 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:18.520442    2583 generic.go:356] "Generic (PLEG): container finished" podID="56811c57-413e-474d-9744-5827780e111d" containerID="1b5de6580c99d3762f1aa6385f2ff30873d699f4f5967e779a8f67cc02708d1f" exitCode=0
Sep 06 10:22:18.520914 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:18.520523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lcn9" event={"ID":"56811c57-413e-474d-9744-5827780e111d","Type":"ContainerDied","Data":"1b5de6580c99d3762f1aa6385f2ff30873d699f4f5967e779a8f67cc02708d1f"}
Sep 06 10:22:19.235059 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:19.235026    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:19.235214 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:19.235140    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:20.234857 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:20.234816    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:20.235247 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:20.234823    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:20.235247 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:20.234929    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:20.235247 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:20.234999    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:20.527849 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:20.527810    2583 generic.go:356] "Generic (PLEG): container finished" podID="56811c57-413e-474d-9744-5827780e111d" containerID="51f8802133e8788f8c088e9a7ac452f083a87dd112c21aa97bb7104942bc795f" exitCode=0
Sep 06 10:22:20.528046 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:20.527868    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lcn9" event={"ID":"56811c57-413e-474d-9744-5827780e111d","Type":"ContainerDied","Data":"51f8802133e8788f8c088e9a7ac452f083a87dd112c21aa97bb7104942bc795f"}
Sep 06 10:22:21.234977 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:21.234779    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:21.235453 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:21.235076    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j5x7" podUID="07743b98-da6f-486a-abff-020a64af3133"
Sep 06 10:22:22.233895 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:22.233858    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:22.234061 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:22.233895    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:22.234061 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.233987    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xpk2b" podUID="18abf90a-c64a-41a5-b6f9-836b95c43600"
Sep 06 10:22:22.234170 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.234085    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ckgmx" podUID="431929b2-d956-4151-acea-b8d910e8c6fc"
Sep 06 10:22:22.898050 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:22.898013    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:22.898510 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.898164    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:22.898510 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.898253    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs podName:18abf90a-c64a-41a5-b6f9-836b95c43600 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:54.898231199 +0000 UTC m=+66.170872680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs") pod "network-metrics-daemon-xpk2b" (UID: "18abf90a-c64a-41a5-b6f9-836b95c43600") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:22:22.998760 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:22.998720    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:22.998951 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:22.998780    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:22.998951 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.998876    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:22.998951 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.998909    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:22:22.998951 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.998928    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:22:22.998951 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.998939    2583 projected.go:196] Error preparing data for projected volume kube-api-access-zvwqh for pod openshift-network-diagnostics/network-check-target-2j5x7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:22.998951 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.998952    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret podName:431929b2-d956-4151-acea-b8d910e8c6fc nodeName:}" failed. No retries permitted until 2026-09-06 10:22:54.998936813 +0000 UTC m=+66.271578296 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret") pod "global-pull-secret-syncer-ckgmx" (UID: "431929b2-d956-4151-acea-b8d910e8c6fc") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:22:22.999267 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:22.998978    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh podName:07743b98-da6f-486a-abff-020a64af3133 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:54.998965669 +0000 UTC m=+66.271607149 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zvwqh" (UniqueName: "kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh") pod "network-check-target-2j5x7" (UID: "07743b98-da6f-486a-abff-020a64af3133") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:22:23.026789 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.026746    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-109.ec2.internal" event="NodeReady"
Sep 06 10:22:23.026970 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.026917    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 06 10:22:23.076924 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.076876    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nb6zp"]
Sep 06 10:22:23.093478 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.093448    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2kcjx"]
Sep 06 10:22:23.094418 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.093858    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nb6zp"
Sep 06 10:22:23.096172 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.095754    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5f7rv\""
Sep 06 10:22:23.096172 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.095950    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 06 10:22:23.096357 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.096276    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 06 10:22:23.096865 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.096552    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 06 10:22:23.109093 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.109051    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c867cc788-k5bm4"]
Sep 06 10:22:23.109250 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.109196    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.111544 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.111515    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mhwb9\""
Sep 06 10:22:23.111675 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.111603    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 06 10:22:23.111675 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.111643    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 06 10:22:23.130910 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.130882    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2kcjx"]
Sep 06 10:22:23.131090 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.130929    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nb6zp"]
Sep 06 10:22:23.131090 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.130951    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c867cc788-k5bm4"]
Sep 06 10:22:23.131090 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.131008    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.133051 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.133028    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 06 10:22:23.133209 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.133175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 06 10:22:23.133273 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.133255    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 06 10:22:23.133326 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.133256    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4td26\""
Sep 06 10:22:23.139242 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.139220    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 06 10:22:23.200265 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200165    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.200265 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200220    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzhzf\" (UniqueName: \"kubernetes.io/projected/ecaebe8d-1299-48ba-80d5-78378304e826-kube-api-access-nzhzf\") pod \"ingress-canary-nb6zp\" (UID: \"ecaebe8d-1299-48ba-80d5-78378304e826\") " pod="openshift-ingress-canary/ingress-canary-nb6zp"
Sep 06 10:22:23.200265 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200242    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbd4k\" (UniqueName: \"kubernetes.io/projected/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-kube-api-access-nbd4k\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.200265 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200263    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-installation-pull-secrets\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.200576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200321    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-image-registry-private-configuration\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.200576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200375    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.200576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200411    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a64550-4561-48a1-8348-a08076259021-ca-trust-extracted\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.200576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-tmp-dir\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.200576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-trusted-ca\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.200576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200482    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-bound-sa-token\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.200576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200502    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8m27\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-kube-api-access-v8m27\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.200576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200528    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-registry-certificates\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.200576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200569    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-config-volume\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.200945 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.200670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecaebe8d-1299-48ba-80d5-78378304e826-cert\") pod \"ingress-canary-nb6zp\" (UID: \"ecaebe8d-1299-48ba-80d5-78378304e826\") " pod="openshift-ingress-canary/ingress-canary-nb6zp"
Sep 06 10:22:23.234085 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.234049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:23.239052 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.239026    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 10:22:23.239550 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.239024    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 10:22:23.239550 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.239486    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xrhk7\""
Sep 06 10:22:23.301954 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.301918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a64550-4561-48a1-8348-a08076259021-ca-trust-extracted\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.301954 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.301957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-tmp-dir\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.302194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.301975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-trusted-ca\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.302194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.301990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-bound-sa-token\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.302194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302008    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8m27\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-kube-api-access-v8m27\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.302194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-registry-certificates\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.302194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-config-volume\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.302194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302118    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecaebe8d-1299-48ba-80d5-78378304e826-cert\") pod \"ingress-canary-nb6zp\" (UID: \"ecaebe8d-1299-48ba-80d5-78378304e826\") " pod="openshift-ingress-canary/ingress-canary-nb6zp"
Sep 06 10:22:23.302194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302151    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.302194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzhzf\" (UniqueName: \"kubernetes.io/projected/ecaebe8d-1299-48ba-80d5-78378304e826-kube-api-access-nzhzf\") pod \"ingress-canary-nb6zp\" (UID: \"ecaebe8d-1299-48ba-80d5-78378304e826\") " pod="openshift-ingress-canary/ingress-canary-nb6zp"
Sep 06 10:22:23.302620 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302210    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbd4k\" (UniqueName: \"kubernetes.io/projected/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-kube-api-access-nbd4k\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.302620 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-installation-pull-secrets\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.302620 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-image-registry-private-configuration\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.302620 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.302620 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302363    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-tmp-dir\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.302620 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302420    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a64550-4561-48a1-8348-a08076259021-ca-trust-extracted\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.302620 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.302470    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:23.302620 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.302534    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls podName:ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:23.802513274 +0000 UTC m=+35.075154768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls") pod "dns-default-2kcjx" (UID: "ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:23.302912 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.302811    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 10:22:23.302912 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.302828    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c867cc788-k5bm4: secret "image-registry-tls" not found
Sep 06 10:22:23.302912 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.302880    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls podName:13a64550-4561-48a1-8348-a08076259021 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:23.802860773 +0000 UTC m=+35.075502283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls") pod "image-registry-6c867cc788-k5bm4" (UID: "13a64550-4561-48a1-8348-a08076259021") : secret "image-registry-tls" not found
Sep 06 10:22:23.303026 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.302966    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-registry-certificates\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.303100 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.303071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-trusted-ca\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.303278 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.303159    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-config-volume\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.306721 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.306694    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-image-registry-private-configuration\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.306817 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.306723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-installation-pull-secrets\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.307093 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.307076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ecaebe8d-1299-48ba-80d5-78378304e826-cert\") pod \"ingress-canary-nb6zp\" (UID: \"ecaebe8d-1299-48ba-80d5-78378304e826\") " pod="openshift-ingress-canary/ingress-canary-nb6zp"
Sep 06 10:22:23.310397 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.310371    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzhzf\" (UniqueName: \"kubernetes.io/projected/ecaebe8d-1299-48ba-80d5-78378304e826-kube-api-access-nzhzf\") pod \"ingress-canary-nb6zp\" (UID: \"ecaebe8d-1299-48ba-80d5-78378304e826\") " pod="openshift-ingress-canary/ingress-canary-nb6zp"
Sep 06 10:22:23.310605 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.310581    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-bound-sa-token\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.310773 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.310754    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbd4k\" (UniqueName: \"kubernetes.io/projected/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-kube-api-access-nbd4k\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.311070 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.311053    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8m27\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-kube-api-access-v8m27\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.407254 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.407218    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nb6zp"
Sep 06 10:22:23.805928 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.805886    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:23.806128 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:23.805999    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:23.806128 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.806077    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 10:22:23.806128 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.806099    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c867cc788-k5bm4: secret "image-registry-tls" not found
Sep 06 10:22:23.806128 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.806113    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:23.806330 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.806162    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls podName:ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:24.806147608 +0000 UTC m=+36.078789095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls") pod "dns-default-2kcjx" (UID: "ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:23.806330 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:23.806177    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls podName:13a64550-4561-48a1-8348-a08076259021 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:24.806169921 +0000 UTC m=+36.078811402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls") pod "image-registry-6c867cc788-k5bm4" (UID: "13a64550-4561-48a1-8348-a08076259021") : secret "image-registry-tls" not found
Sep 06 10:22:24.234571 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:24.234538    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:24.235130 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:24.234576    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:24.236781 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:24.236753    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 10:22:24.236916 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:24.236823    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xmc8p\""
Sep 06 10:22:24.236916 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:24.236861    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 10:22:24.776415 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:24.776380    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nb6zp"]
Sep 06 10:22:24.780640 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:22:24.780617    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecaebe8d_1299_48ba_80d5_78378304e826.slice/crio-324c5cf992d01fb8d4e17072c88e20f16e47b5b636c3297f444b34ad5cb2002a WatchSource:0}: Error finding container 324c5cf992d01fb8d4e17072c88e20f16e47b5b636c3297f444b34ad5cb2002a: Status 404 returned error can't find the container with id 324c5cf992d01fb8d4e17072c88e20f16e47b5b636c3297f444b34ad5cb2002a
Sep 06 10:22:24.813590 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:24.813411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:24.813755 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:24.813563    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:24.813755 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:24.813637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:24.813755 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:24.813678    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls podName:ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:26.813658894 +0000 UTC m=+38.086300387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls") pod "dns-default-2kcjx" (UID: "ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:24.813755 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:24.813741    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 10:22:24.813755 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:24.813752    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c867cc788-k5bm4: secret "image-registry-tls" not found
Sep 06 10:22:24.813917 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:24.813797    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls podName:13a64550-4561-48a1-8348-a08076259021 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:26.81378562 +0000 UTC m=+38.086427115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls") pod "image-registry-6c867cc788-k5bm4" (UID: "13a64550-4561-48a1-8348-a08076259021") : secret "image-registry-tls" not found
Sep 06 10:22:25.540944 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:25.540908    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nb6zp" event={"ID":"ecaebe8d-1299-48ba-80d5-78378304e826","Type":"ContainerStarted","Data":"324c5cf992d01fb8d4e17072c88e20f16e47b5b636c3297f444b34ad5cb2002a"}
Sep 06 10:22:25.543957 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:25.543923    2583 generic.go:356] "Generic (PLEG): container finished" podID="56811c57-413e-474d-9744-5827780e111d" containerID="a2ad4b43fc9224df495cc69bd51bdcb1ed5536e9f36d0abd86f63137b856f04d" exitCode=0
Sep 06 10:22:25.544069 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:25.543987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lcn9" event={"ID":"56811c57-413e-474d-9744-5827780e111d","Type":"ContainerDied","Data":"a2ad4b43fc9224df495cc69bd51bdcb1ed5536e9f36d0abd86f63137b856f04d"}
Sep 06 10:22:26.550233 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:26.550196    2583 generic.go:356] "Generic (PLEG): container finished" podID="56811c57-413e-474d-9744-5827780e111d" containerID="ae54f44d9922eefd0036f18699935a4f4da84f15cde952633059863898217546" exitCode=0
Sep 06 10:22:26.550662 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:26.550271    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lcn9" event={"ID":"56811c57-413e-474d-9744-5827780e111d","Type":"ContainerDied","Data":"ae54f44d9922eefd0036f18699935a4f4da84f15cde952633059863898217546"}
Sep 06 10:22:26.828544 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:26.828460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:26.828544 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:26.828525    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:26.828753 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:26.828646    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:26.828753 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:26.828697    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 10:22:26.828753 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:26.828719    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c867cc788-k5bm4: secret "image-registry-tls" not found
Sep 06 10:22:26.828753 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:26.828727    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls podName:ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:30.828705682 +0000 UTC m=+42.101347164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls") pod "dns-default-2kcjx" (UID: "ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:26.828966 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:26.828761    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls podName:13a64550-4561-48a1-8348-a08076259021 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:30.828748502 +0000 UTC m=+42.101389982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls") pod "image-registry-6c867cc788-k5bm4" (UID: "13a64550-4561-48a1-8348-a08076259021") : secret "image-registry-tls" not found
Sep 06 10:22:27.556388 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:27.556351    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5lcn9" event={"ID":"56811c57-413e-474d-9744-5827780e111d","Type":"ContainerStarted","Data":"f41c4442637f6c371d2839bfb2a4dbbf494cd315698732b7fa12f7fbe9d0f080"}
Sep 06 10:22:27.557814 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:27.557788    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nb6zp" event={"ID":"ecaebe8d-1299-48ba-80d5-78378304e826","Type":"ContainerStarted","Data":"931a29472e041b1750a93b7e902355d3ec8e3137f748b2840bd78c8c78efdeee"}
Sep 06 10:22:27.580096 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:27.580036    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5lcn9" podStartSLOduration=9.557298958 podStartE2EDuration="38.580020554s" totalImagesPullingTime="29.022721596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:27.579108398 +0000 UTC m=+38.851749912" watchObservedRunningTime="2026-09-06 10:22:27.580020554 +0000 UTC m=+38.852662058"
Sep 06 10:22:27.593698 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:27.593644    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nb6zp" podStartSLOduration=2.106170277 podStartE2EDuration="4.593624546s" totalImagesPullingTime="2.487454269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:27.593068063 +0000 UTC m=+38.865709565" watchObservedRunningTime="2026-09-06 10:22:27.593624546 +0000 UTC m=+38.866266050"
Sep 06 10:22:30.859618 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:30.859566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:30.860010 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:30.859658    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:30.860010 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:30.859742    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:30.860010 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:30.859742    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 10:22:30.860010 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:30.859813    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c867cc788-k5bm4: secret "image-registry-tls" not found
Sep 06 10:22:30.860010 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:30.859793    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls podName:ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:38.859778484 +0000 UTC m=+50.132419986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls") pod "dns-default-2kcjx" (UID: "ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:30.860010 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:30.859881    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls podName:13a64550-4561-48a1-8348-a08076259021 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:38.859866857 +0000 UTC m=+50.132508337 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls") pod "image-registry-6c867cc788-k5bm4" (UID: "13a64550-4561-48a1-8348-a08076259021") : secret "image-registry-tls" not found
Sep 06 10:22:33.663682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.663645    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-48hq5"]
Sep 06 10:22:33.715305 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.715270    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48hq5"]
Sep 06 10:22:33.715513 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.715423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.717712 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.717689    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 06 10:22:33.719448 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.719426    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 06 10:22:33.719448 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.719439    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 06 10:22:33.719614 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.719436    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p462p\""
Sep 06 10:22:33.719614 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.719569    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 06 10:22:33.885766 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.885725    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bd8\" (UniqueName: \"kubernetes.io/projected/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-api-access-x9bd8\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.885766 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.885769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.885981 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.885792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c438715-ccf5-4929-b41d-501ead22ee0a-crio-socket\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.885981 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.885807    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.885981 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.885865    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c438715-ccf5-4929-b41d-501ead22ee0a-data-volume\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.986988 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.986892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bd8\" (UniqueName: \"kubernetes.io/projected/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-api-access-x9bd8\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.986988 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.986937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.986988 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.986958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c438715-ccf5-4929-b41d-501ead22ee0a-crio-socket\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.987217 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:33.987068    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:33.987217 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.987125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3c438715-ccf5-4929-b41d-501ead22ee0a-crio-socket\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.987217 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:33.987135    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 10:22:33.987217 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.987080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.987217 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:33.987141    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls podName:3c438715-ccf5-4929-b41d-501ead22ee0a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:34.487121289 +0000 UTC m=+45.759762774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48hq5" (UID: "3c438715-ccf5-4929-b41d-501ead22ee0a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:33.987217 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:33.987203    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm podName:3c438715-ccf5-4929-b41d-501ead22ee0a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:34.487189189 +0000 UTC m=+45.759830672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48hq5" (UID: "3c438715-ccf5-4929-b41d-501ead22ee0a") : configmap "kube-rbac-proxy" not found
Sep 06 10:22:33.987507 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.987235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c438715-ccf5-4929-b41d-501ead22ee0a-data-volume\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.987645 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.987626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c438715-ccf5-4929-b41d-501ead22ee0a-data-volume\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:33.996951 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:33.996923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bd8\" (UniqueName: \"kubernetes.io/projected/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-api-access-x9bd8\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:34.492040 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:34.492002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:34.492040 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:34.492042    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:34.492264 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:34.492120    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 10:22:34.492264 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:34.492153    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:34.492264 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:34.492170    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm podName:3c438715-ccf5-4929-b41d-501ead22ee0a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:35.49215803 +0000 UTC m=+46.764799512 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48hq5" (UID: "3c438715-ccf5-4929-b41d-501ead22ee0a") : configmap "kube-rbac-proxy" not found
Sep 06 10:22:34.492264 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:34.492223    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls podName:3c438715-ccf5-4929-b41d-501ead22ee0a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:35.492209685 +0000 UTC m=+46.764851167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48hq5" (UID: "3c438715-ccf5-4929-b41d-501ead22ee0a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:35.416756 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:35.416728    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jcbhz_15b6b3de-8170-431a-b270-81ebb6c5f7e9/dns-node-resolver/0.log"
Sep 06 10:22:35.498513 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:35.498408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:35.498513 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:35.498461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:35.498750 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:35.498543    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 10:22:35.498750 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:35.498563    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:35.498750 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:35.498614    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm podName:3c438715-ccf5-4929-b41d-501ead22ee0a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:37.498595292 +0000 UTC m=+48.771236773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48hq5" (UID: "3c438715-ccf5-4929-b41d-501ead22ee0a") : configmap "kube-rbac-proxy" not found
Sep 06 10:22:35.498750 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:35.498632    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls podName:3c438715-ccf5-4929-b41d-501ead22ee0a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:37.498623644 +0000 UTC m=+48.771265125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48hq5" (UID: "3c438715-ccf5-4929-b41d-501ead22ee0a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:36.617549 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:36.617521    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vd727_9b4c8743-d239-4ff4-97d3-55e1e861cc1e/node-ca/0.log"
Sep 06 10:22:37.512295 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:37.512252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:37.512493 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:37.512301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:37.512493 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:37.512428    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:37.512493 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:37.512478    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 06 10:22:37.512589 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:37.512489    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls podName:3c438715-ccf5-4929-b41d-501ead22ee0a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:41.512474844 +0000 UTC m=+52.785116330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48hq5" (UID: "3c438715-ccf5-4929-b41d-501ead22ee0a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:37.512635 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:37.512599    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm podName:3c438715-ccf5-4929-b41d-501ead22ee0a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:41.51258442 +0000 UTC m=+52.785225901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-48hq5" (UID: "3c438715-ccf5-4929-b41d-501ead22ee0a") : configmap "kube-rbac-proxy" not found
Sep 06 10:22:37.617444 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:37.617394    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nb6zp_ecaebe8d-1299-48ba-80d5-78378304e826/serve-healthcheck-canary/0.log"
Sep 06 10:22:38.923315 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:38.923274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:38.923699 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:38.923353    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:38.923699 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:38.923450    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 06 10:22:38.923699 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:38.923455    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 06 10:22:38.923699 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:38.923476    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c867cc788-k5bm4: secret "image-registry-tls" not found
Sep 06 10:22:38.923699 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:38.923509    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls podName:ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:54.923492942 +0000 UTC m=+66.196134429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls") pod "dns-default-2kcjx" (UID: "ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1") : secret "dns-default-metrics-tls" not found
Sep 06 10:22:38.923699 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:38.923532    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls podName:13a64550-4561-48a1-8348-a08076259021 nodeName:}" failed. No retries permitted until 2026-09-06 10:22:54.923518917 +0000 UTC m=+66.196160398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls") pod "image-registry-6c867cc788-k5bm4" (UID: "13a64550-4561-48a1-8348-a08076259021") : secret "image-registry-tls" not found
Sep 06 10:22:41.545420 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:41.545381    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:41.545420 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:41.545425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:41.545856 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:41.545538    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:41.545856 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:22:41.545611    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls podName:3c438715-ccf5-4929-b41d-501ead22ee0a nodeName:}" failed. No retries permitted until 2026-09-06 10:22:49.545590785 +0000 UTC m=+60.818232280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-48hq5" (UID: "3c438715-ccf5-4929-b41d-501ead22ee0a") : secret "insights-runtime-extractor-tls" not found
Sep 06 10:22:41.546098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:41.546078    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3c438715-ccf5-4929-b41d-501ead22ee0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:49.539086 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:49.539059    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fcsrt"
Sep 06 10:22:49.607716 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:49.607670    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:49.610035 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:49.610008    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3c438715-ccf5-4929-b41d-501ead22ee0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-48hq5\" (UID: \"3c438715-ccf5-4929-b41d-501ead22ee0a\") " pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:49.626781 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:49.626747    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p462p\""
Sep 06 10:22:49.636120 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:49.636096    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-48hq5"
Sep 06 10:22:49.927843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:49.927809    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-48hq5"]
Sep 06 10:22:49.931467 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:22:49.931441    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c438715_ccf5_4929_b41d_501ead22ee0a.slice/crio-990ace875cf5a4d70751b82f6e76c511831e0f3075eab59afc3e5d4aa3d25040 WatchSource:0}: Error finding container 990ace875cf5a4d70751b82f6e76c511831e0f3075eab59afc3e5d4aa3d25040: Status 404 returned error can't find the container with id 990ace875cf5a4d70751b82f6e76c511831e0f3075eab59afc3e5d4aa3d25040
Sep 06 10:22:50.611297 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:50.611260    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48hq5" event={"ID":"3c438715-ccf5-4929-b41d-501ead22ee0a","Type":"ContainerStarted","Data":"b275b2252dc5152c8c0fe6a22d35562ff7aa043ec7097b1d64c0002e32965fed"}
Sep 06 10:22:50.611661 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:50.611318    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48hq5" event={"ID":"3c438715-ccf5-4929-b41d-501ead22ee0a","Type":"ContainerStarted","Data":"990ace875cf5a4d70751b82f6e76c511831e0f3075eab59afc3e5d4aa3d25040"}
Sep 06 10:22:51.615500 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:51.615456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48hq5" event={"ID":"3c438715-ccf5-4929-b41d-501ead22ee0a","Type":"ContainerStarted","Data":"75567360a6aa0410988337a9ac47c190f7068a2f5c70044172270dea5c2696cc"}
Sep 06 10:22:52.619885 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:52.619854    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-48hq5" event={"ID":"3c438715-ccf5-4929-b41d-501ead22ee0a","Type":"ContainerStarted","Data":"c4e7a25075d1be04df55f69d23e8649055534c886c133b7bbc5cc442f874938a"}
Sep 06 10:22:52.637788 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:52.637739    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-48hq5" podStartSLOduration=17.762743742 podStartE2EDuration="19.637723205s" totalImagesPullingTime="1.874979463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:22:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:52.637157272 +0000 UTC m=+63.909798775" watchObservedRunningTime="2026-09-06 10:22:52.637723205 +0000 UTC m=+63.910364707"
Sep 06 10:22:54.952843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:54.952804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:54.953264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:54.952861    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:54.953264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:54.952888    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:54.955217 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:54.955182    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 10:22:54.955217 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:54.955214    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1-metrics-tls\") pod \"dns-default-2kcjx\" (UID: \"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1\") " pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:54.955426 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:54.955282    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"image-registry-6c867cc788-k5bm4\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") " pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:54.965882 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:54.965849    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18abf90a-c64a-41a5-b6f9-836b95c43600-metrics-certs\") pod \"network-metrics-daemon-xpk2b\" (UID: \"18abf90a-c64a-41a5-b6f9-836b95c43600\") " pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:55.054119 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.054078    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:55.054306 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.054146    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:55.056258 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.056238    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 10:22:55.056258 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.056252    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 10:22:55.066420 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.066394    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 10:22:55.067083 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.067062    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/431929b2-d956-4151-acea-b8d910e8c6fc-original-pull-secret\") pod \"global-pull-secret-syncer-ckgmx\" (UID: \"431929b2-d956-4151-acea-b8d910e8c6fc\") " pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:55.077311 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.077278    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvwqh\" (UniqueName: \"kubernetes.io/projected/07743b98-da6f-486a-abff-020a64af3133-kube-api-access-zvwqh\") pod \"network-check-target-2j5x7\" (UID: \"07743b98-da6f-486a-abff-020a64af3133\") " pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:55.148922 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.148888    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xmc8p\""
Sep 06 10:22:55.153487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.153467    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ckgmx"
Sep 06 10:22:55.157789 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.157768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xpk2b"
Sep 06 10:22:55.222470 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.222428    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mhwb9\""
Sep 06 10:22:55.230441 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.230409    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:22:55.242727 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.242696    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4td26\""
Sep 06 10:22:55.251366 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.251308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:55.348794 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.348758    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xrhk7\""
Sep 06 10:22:55.357624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.357596    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:22:55.420417 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.420386    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ckgmx"]
Sep 06 10:22:55.424710 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:22:55.424678    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431929b2_d956_4151_acea_b8d910e8c6fc.slice/crio-14af6997b2ca987ad2e6d6cbb2745e497a7df8f0346f25728972edaaafb09c65 WatchSource:0}: Error finding container 14af6997b2ca987ad2e6d6cbb2745e497a7df8f0346f25728972edaaafb09c65: Status 404 returned error can't find the container with id 14af6997b2ca987ad2e6d6cbb2745e497a7df8f0346f25728972edaaafb09c65
Sep 06 10:22:55.426368 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.426324    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xpk2b"]
Sep 06 10:22:55.429490 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:22:55.429463    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18abf90a_c64a_41a5_b6f9_836b95c43600.slice/crio-db679a850f8a49b251d67ce9c1cb98e6d38fc6c03f80fc359d7e251838a30221 WatchSource:0}: Error finding container db679a850f8a49b251d67ce9c1cb98e6d38fc6c03f80fc359d7e251838a30221: Status 404 returned error can't find the container with id db679a850f8a49b251d67ce9c1cb98e6d38fc6c03f80fc359d7e251838a30221
Sep 06 10:22:55.505220 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.505139    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2kcjx"]
Sep 06 10:22:55.508363 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:22:55.508324    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2b5a90_8d9e_4821_a415_f2f0ffc7c8f1.slice/crio-119dff3ca998aca61497a455bed7d81b9a7476b9c1ce98ee0f323ce0d9b1bc56 WatchSource:0}: Error finding container 119dff3ca998aca61497a455bed7d81b9a7476b9c1ce98ee0f323ce0d9b1bc56: Status 404 returned error can't find the container with id 119dff3ca998aca61497a455bed7d81b9a7476b9c1ce98ee0f323ce0d9b1bc56
Sep 06 10:22:55.530454 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.530405    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c867cc788-k5bm4"]
Sep 06 10:22:55.534134 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:22:55.534108    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a64550_4561_48a1_8348_a08076259021.slice/crio-b43d7bf432cee8bbfdaa0ed920803c8bfdef2e022926c89ad6913615672cb295 WatchSource:0}: Error finding container b43d7bf432cee8bbfdaa0ed920803c8bfdef2e022926c89ad6913615672cb295: Status 404 returned error can't find the container with id b43d7bf432cee8bbfdaa0ed920803c8bfdef2e022926c89ad6913615672cb295
Sep 06 10:22:55.616043 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.615589    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"]
Sep 06 10:22:55.617572 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.617543    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"]
Sep 06 10:22:55.617771 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.617749    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.620082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.619862    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 06 10:22:55.620082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.620030    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 06 10:22:55.622051 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.620298    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 06 10:22:55.622051 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.620311    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 06 10:22:55.624540 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.624515    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.626265 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.626245    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 06 10:22:55.627646 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.626999    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 06 10:22:55.627646 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.627426    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 06 10:22:55.627833 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.627717    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 06 10:22:55.628447 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.628421    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"]
Sep 06 10:22:55.629127 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.629092    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xpk2b" event={"ID":"18abf90a-c64a-41a5-b6f9-836b95c43600","Type":"ContainerStarted","Data":"db679a850f8a49b251d67ce9c1cb98e6d38fc6c03f80fc359d7e251838a30221"}
Sep 06 10:22:55.631750 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.631314    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2j5x7"]
Sep 06 10:22:55.631833 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.631809    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"]
Sep 06 10:22:55.632404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.632370    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ckgmx" event={"ID":"431929b2-d956-4151-acea-b8d910e8c6fc","Type":"ContainerStarted","Data":"14af6997b2ca987ad2e6d6cbb2745e497a7df8f0346f25728972edaaafb09c65"}
Sep 06 10:22:55.634526 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.634495    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4" event={"ID":"13a64550-4561-48a1-8348-a08076259021","Type":"ContainerStarted","Data":"9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0"}
Sep 06 10:22:55.634645 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.634545    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4" event={"ID":"13a64550-4561-48a1-8348-a08076259021","Type":"ContainerStarted","Data":"b43d7bf432cee8bbfdaa0ed920803c8bfdef2e022926c89ad6913615672cb295"}
Sep 06 10:22:55.636773 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.635052    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:22:55.636773 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:22:55.636059    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07743b98_da6f_486a_abff_020a64af3133.slice/crio-d8d99de6dd41283f1d425e61a81dde80b0896de41c993452cc890fdd304b1867 WatchSource:0}: Error finding container d8d99de6dd41283f1d425e61a81dde80b0896de41c993452cc890fdd304b1867: Status 404 returned error can't find the container with id d8d99de6dd41283f1d425e61a81dde80b0896de41c993452cc890fdd304b1867
Sep 06 10:22:55.636773 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.636597    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2kcjx" event={"ID":"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1","Type":"ContainerStarted","Data":"119dff3ca998aca61497a455bed7d81b9a7476b9c1ce98ee0f323ce0d9b1bc56"}
Sep 06 10:22:55.657845 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.657805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-ca\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.657845 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.657848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-hub\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.658066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.657878    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.658066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.657942    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.658066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.657974    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.658066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.658061    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39fdbbe4-f680-4373-933f-17b5bceaec41-tmp\") pod \"klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc\" (UID: \"39fdbbe4-f680-4373-933f-17b5bceaec41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.658243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.658116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39fdbbe4-f680-4373-933f-17b5bceaec41-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc\" (UID: \"39fdbbe4-f680-4373-933f-17b5bceaec41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.658243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.658147    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwhx\" (UniqueName: \"kubernetes.io/projected/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-kube-api-access-4hwhx\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.658468 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.658421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qplm\" (UniqueName: \"kubernetes.io/projected/39fdbbe4-f680-4373-933f-17b5bceaec41-kube-api-access-9qplm\") pod \"klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc\" (UID: \"39fdbbe4-f680-4373-933f-17b5bceaec41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.671385 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.671325    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4" podStartSLOduration=47.671306357 podStartE2EDuration="47.671306357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:55.670107412 +0000 UTC m=+66.942748914" watchObservedRunningTime="2026-09-06 10:22:55.671306357 +0000 UTC m=+66.943947863"
Sep 06 10:22:55.704396 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.704361    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c867cc788-k5bm4"]
Sep 06 10:22:55.759714 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.759627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qplm\" (UniqueName: \"kubernetes.io/projected/39fdbbe4-f680-4373-933f-17b5bceaec41-kube-api-access-9qplm\") pod \"klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc\" (UID: \"39fdbbe4-f680-4373-933f-17b5bceaec41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.759714 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.759688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-ca\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.759948 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.759722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-hub\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.760008 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.759936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.760066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.760043    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.760180 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.760074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.760180 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.760172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39fdbbe4-f680-4373-933f-17b5bceaec41-tmp\") pod \"klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc\" (UID: \"39fdbbe4-f680-4373-933f-17b5bceaec41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.760309 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.760197    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39fdbbe4-f680-4373-933f-17b5bceaec41-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc\" (UID: \"39fdbbe4-f680-4373-933f-17b5bceaec41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.760309 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.760224    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwhx\" (UniqueName: \"kubernetes.io/projected/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-kube-api-access-4hwhx\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.760672 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.760616    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39fdbbe4-f680-4373-933f-17b5bceaec41-tmp\") pod \"klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc\" (UID: \"39fdbbe4-f680-4373-933f-17b5bceaec41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.760873 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.760726    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.763673 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.763639    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.763790 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.763767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-hub\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.763836 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.763804    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/39fdbbe4-f680-4373-933f-17b5bceaec41-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc\" (UID: \"39fdbbe4-f680-4373-933f-17b5bceaec41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.764264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.764237    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.765111 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.765087    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-ca\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.784669 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.784629    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qplm\" (UniqueName: \"kubernetes.io/projected/39fdbbe4-f680-4373-933f-17b5bceaec41-kube-api-access-9qplm\") pod \"klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc\" (UID: \"39fdbbe4-f680-4373-933f-17b5bceaec41\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.784669 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.784651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwhx\" (UniqueName: \"kubernetes.io/projected/9e5c586c-71ff-4e9c-817b-78e1487ffcf0-kube-api-access-4hwhx\") pod \"cluster-proxy-proxy-agent-7dffff9cb4-rnt4l\" (UID: \"9e5c586c-71ff-4e9c-817b-78e1487ffcf0\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:55.934220 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.934179    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:22:55.951159 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:55.951130    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"
Sep 06 10:22:56.229964 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:56.229928    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"]
Sep 06 10:22:56.234904 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:22:56.234865    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fdbbe4_f680_4373_933f_17b5bceaec41.slice/crio-5f9c6cd76ce1706b544e3a1f59fa5acaca2bb7df6ed2f96f1b43e9786ba02e32 WatchSource:0}: Error finding container 5f9c6cd76ce1706b544e3a1f59fa5acaca2bb7df6ed2f96f1b43e9786ba02e32: Status 404 returned error can't find the container with id 5f9c6cd76ce1706b544e3a1f59fa5acaca2bb7df6ed2f96f1b43e9786ba02e32
Sep 06 10:22:56.263907 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:56.263796    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l"]
Sep 06 10:22:56.642306 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:56.642267    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc" event={"ID":"39fdbbe4-f680-4373-933f-17b5bceaec41","Type":"ContainerStarted","Data":"5f9c6cd76ce1706b544e3a1f59fa5acaca2bb7df6ed2f96f1b43e9786ba02e32"}
Sep 06 10:22:56.644077 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:56.644048    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2j5x7" event={"ID":"07743b98-da6f-486a-abff-020a64af3133","Type":"ContainerStarted","Data":"d8d99de6dd41283f1d425e61a81dde80b0896de41c993452cc890fdd304b1867"}
Sep 06 10:22:56.680570 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:22:56.680523    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e5c586c_71ff_4e9c_817b_78e1487ffcf0.slice/crio-efe6d5ac34127376b0c7efe9414096203b7830867fc05bfc6af68f954048d516 WatchSource:0}: Error finding container efe6d5ac34127376b0c7efe9414096203b7830867fc05bfc6af68f954048d516: Status 404 returned error can't find the container with id efe6d5ac34127376b0c7efe9414096203b7830867fc05bfc6af68f954048d516
Sep 06 10:22:57.650275 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:57.650225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l" event={"ID":"9e5c586c-71ff-4e9c-817b-78e1487ffcf0","Type":"ContainerStarted","Data":"efe6d5ac34127376b0c7efe9414096203b7830867fc05bfc6af68f954048d516"}
Sep 06 10:22:58.660843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:58.660693    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2kcjx" event={"ID":"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1","Type":"ContainerStarted","Data":"c3ce599afc44f9e14afc548bf2e2aac6cdfca82203448c93e25e0ff8ae706810"}
Sep 06 10:22:58.664890 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:58.664083    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xpk2b" event={"ID":"18abf90a-c64a-41a5-b6f9-836b95c43600","Type":"ContainerStarted","Data":"97c07b5077033a2962bc6bafaf7c19a72f30553cfe837af36faa481c6a0aa73a"}
Sep 06 10:22:58.664890 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:58.664132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xpk2b" event={"ID":"18abf90a-c64a-41a5-b6f9-836b95c43600","Type":"ContainerStarted","Data":"4e036bb1ed8d592214397aca1d3b3893d5637303b0ac95d610b5536e57d747b1"}
Sep 06 10:22:58.680628 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:22:58.680549    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xpk2b" podStartSLOduration=67.333548655 podStartE2EDuration="1m9.680527281s" totalImagesPullingTime="2.346978626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:22:58.67932805 +0000 UTC m=+69.951969552" watchObservedRunningTime="2026-09-06 10:22:58.680527281 +0000 UTC m=+69.953168785"
Sep 06 10:23:05.695897 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.695860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ckgmx" event={"ID":"431929b2-d956-4151-acea-b8d910e8c6fc","Type":"ContainerStarted","Data":"0724c7fd6542f485cf7b99701ef256e4e0b77f177610865506c5097306c513e5"}
Sep 06 10:23:05.697346 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.697304    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2j5x7" event={"ID":"07743b98-da6f-486a-abff-020a64af3133","Type":"ContainerStarted","Data":"7fd5b95fd4e0f729966be30aa50dc377cb3f12cee5f94860d613775e90bd378d"}
Sep 06 10:23:05.697470 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.697378    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:23:05.699413 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.699382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l" event={"ID":"9e5c586c-71ff-4e9c-817b-78e1487ffcf0","Type":"ContainerStarted","Data":"d42e29913ceebcbe66b4c6315b46e8677da310451d48a29ead93e6d9cc18fe89"}
Sep 06 10:23:05.699542 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.699420    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l" event={"ID":"9e5c586c-71ff-4e9c-817b-78e1487ffcf0","Type":"ContainerStarted","Data":"d06d5b2d05777258e854872d6a3fb7265147385a79b45876ae269459df57f74e"}
Sep 06 10:23:05.699542 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.699434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l" event={"ID":"9e5c586c-71ff-4e9c-817b-78e1487ffcf0","Type":"ContainerStarted","Data":"d74c88f1e979fc663e5b3c28fd7c7dfd0b5e3f0110b1b16a1b1fe780eda93613"}
Sep 06 10:23:05.701007 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.700980    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2kcjx" event={"ID":"ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1","Type":"ContainerStarted","Data":"431da39e3bfcae4242e4efea21442b7ed04576b21877f3f775758a0d367f8ae5"}
Sep 06 10:23:05.701087 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.701037    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:23:05.702348 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.702311    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc" event={"ID":"39fdbbe4-f680-4373-933f-17b5bceaec41","Type":"ContainerStarted","Data":"290d29efa89163284a8f52068f3a4d4892c8e9309b6162aa6bdbf49eb2fce8f9"}
Sep 06 10:23:05.702514 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.702498    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:23:05.704158 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.704139    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc"
Sep 06 10:23:05.709675 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.709630    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ckgmx" podStartSLOduration=67.032919694 podStartE2EDuration="1m16.709596758s" totalImagesPullingTime="9.676677064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:05.709287353 +0000 UTC m=+76.981928856" watchObservedRunningTime="2026-09-06 10:23:05.709596758 +0000 UTC m=+76.982238262"
Sep 06 10:23:05.727419 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.727377    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2kcjx" podStartSLOduration=40.124044388 podStartE2EDuration="42.727359204s" totalImagesPullingTime="2.603314816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:05.726139715 +0000 UTC m=+76.998781219" watchObservedRunningTime="2026-09-06 10:23:05.727359204 +0000 UTC m=+77.000000707"
Sep 06 10:23:05.742271 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.742213    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dc9b8cf9d-zh8bc" podStartSLOduration=1.8536266129999999 podStartE2EDuration="10.742194489s" totalImagesPullingTime="8.888567876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:05.741504585 +0000 UTC m=+77.014146088" watchObservedRunningTime="2026-09-06 10:23:05.742194489 +0000 UTC m=+77.014835991"
Sep 06 10:23:05.761427 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.761370    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2j5x7" podStartSLOduration=67.303498513 podStartE2EDuration="1m16.761355159s" totalImagesPullingTime="9.457856646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:05.760625011 +0000 UTC m=+77.033266514" watchObservedRunningTime="2026-09-06 10:23:05.761355159 +0000 UTC m=+77.033996659"
Sep 06 10:23:05.777714 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.777667    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dffff9cb4-rnt4l" podStartSLOduration=2.334364721 podStartE2EDuration="10.777649209s" totalImagesPullingTime="8.443284488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:22:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:05.776864899 +0000 UTC m=+77.049506403" watchObservedRunningTime="2026-09-06 10:23:05.777649209 +0000 UTC m=+77.050290712"
Sep 06 10:23:05.925105 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.925074    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-75nqd"]
Sep 06 10:23:05.928173 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.928157    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:05.930699 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.930670    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 06 10:23:05.930803 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.930698    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 06 10:23:05.930803 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.930754    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 06 10:23:05.930803 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.930672    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 06 10:23:05.931041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.930875    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 06 10:23:05.931041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.930949    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lcgqx\""
Sep 06 10:23:05.936684 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:05.936662    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-75nqd"]
Sep 06 10:23:06.041642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.041533    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da6ae229-5fa0-401a-b78f-893f261ba735-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.041642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.041608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da6ae229-5fa0-401a-b78f-893f261ba735-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.041819 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.041680    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmbv\" (UniqueName: \"kubernetes.io/projected/da6ae229-5fa0-401a-b78f-893f261ba735-kube-api-access-hdmbv\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.041819 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.041741    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6ae229-5fa0-401a-b78f-893f261ba735-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.143089 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.143044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da6ae229-5fa0-401a-b78f-893f261ba735-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.143089 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.143089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmbv\" (UniqueName: \"kubernetes.io/projected/da6ae229-5fa0-401a-b78f-893f261ba735-kube-api-access-hdmbv\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.143323 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.143277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6ae229-5fa0-401a-b78f-893f261ba735-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.143411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.143377    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da6ae229-5fa0-401a-b78f-893f261ba735-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.143469 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:23:06.143413    2583 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 06 10:23:06.143518 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:23:06.143496    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6ae229-5fa0-401a-b78f-893f261ba735-prometheus-operator-tls podName:da6ae229-5fa0-401a-b78f-893f261ba735 nodeName:}" failed. No retries permitted until 2026-09-06 10:23:06.643476111 +0000 UTC m=+77.916117592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/da6ae229-5fa0-401a-b78f-893f261ba735-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-75nqd" (UID: "da6ae229-5fa0-401a-b78f-893f261ba735") : secret "prometheus-operator-tls" not found
Sep 06 10:23:06.144008 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.143990    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da6ae229-5fa0-401a-b78f-893f261ba735-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.145442 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.145424    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da6ae229-5fa0-401a-b78f-893f261ba735-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.151325 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.151302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmbv\" (UniqueName: \"kubernetes.io/projected/da6ae229-5fa0-401a-b78f-893f261ba735-kube-api-access-hdmbv\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.647374 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.647299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6ae229-5fa0-401a-b78f-893f261ba735-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.649790 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.649765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/da6ae229-5fa0-401a-b78f-893f261ba735-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-75nqd\" (UID: \"da6ae229-5fa0-401a-b78f-893f261ba735\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:06.838081 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:06.838028    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd"
Sep 06 10:23:07.108476 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:07.108446    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-75nqd"]
Sep 06 10:23:07.111349 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:23:07.111301    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6ae229_5fa0_401a_b78f_893f261ba735.slice/crio-087098f6543b84b22f391d9403d18dc7509fde796b11cbf7d8980eaf7f6e89d7 WatchSource:0}: Error finding container 087098f6543b84b22f391d9403d18dc7509fde796b11cbf7d8980eaf7f6e89d7: Status 404 returned error can't find the container with id 087098f6543b84b22f391d9403d18dc7509fde796b11cbf7d8980eaf7f6e89d7
Sep 06 10:23:07.710787 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:07.710748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd" event={"ID":"da6ae229-5fa0-401a-b78f-893f261ba735","Type":"ContainerStarted","Data":"087098f6543b84b22f391d9403d18dc7509fde796b11cbf7d8980eaf7f6e89d7"}
Sep 06 10:23:08.715656 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:08.715623    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd" event={"ID":"da6ae229-5fa0-401a-b78f-893f261ba735","Type":"ContainerStarted","Data":"c8cc87eadef97075d5649739fdd8d9e8a97dc5239fd2359ba858e30b90297c5a"}
Sep 06 10:23:08.716008 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:08.715699    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd" event={"ID":"da6ae229-5fa0-401a-b78f-893f261ba735","Type":"ContainerStarted","Data":"ae836bb32febc1eea4553df94bec0f6a8996f6f06dae391558458d917900b5cb"}
Sep 06 10:23:08.732207 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:08.732147    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-75nqd" podStartSLOduration=2.680100084 podStartE2EDuration="3.732125377s" totalImagesPullingTime="1.052025293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:08.73085163 +0000 UTC m=+80.003493135" watchObservedRunningTime="2026-09-06 10:23:08.732125377 +0000 UTC m=+80.004766881"
Sep 06 10:23:09.708105 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:09.708076    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2kcjx"
Sep 06 10:23:10.319849 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.319813    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2x2n2"]
Sep 06 10:23:10.322746 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.322728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.325156 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.325129    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 06 10:23:10.325293 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.325157    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 06 10:23:10.325293 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.325164    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 06 10:23:10.325293 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.325210    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fsv89\""
Sep 06 10:23:10.376817 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.376774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62085ba7-380b-476e-b255-be186961cedf-metrics-client-ca\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.376817 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.376820    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/62085ba7-380b-476e-b255-be186961cedf-node-exporter-accelerators-collector-config\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.377064 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.376900    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/62085ba7-380b-476e-b255-be186961cedf-node-exporter-wtmp\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.377064 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.376966    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-tls\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.377064 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.376993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/62085ba7-380b-476e-b255-be186961cedf-node-exporter-textfile\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.377211 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.377069    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/62085ba7-380b-476e-b255-be186961cedf-root\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.377211 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.377103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62085ba7-380b-476e-b255-be186961cedf-sys\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.377211 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.377146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmjh\" (UniqueName: \"kubernetes.io/projected/62085ba7-380b-476e-b255-be186961cedf-kube-api-access-2nmjh\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.377211 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.377184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478547 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478488    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/62085ba7-380b-476e-b255-be186961cedf-node-exporter-wtmp\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478547 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478546    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-tls\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478547 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/62085ba7-380b-476e-b255-be186961cedf-node-exporter-textfile\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478591    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/62085ba7-380b-476e-b255-be186961cedf-root\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62085ba7-380b-476e-b255-be186961cedf-sys\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmjh\" (UniqueName: \"kubernetes.io/projected/62085ba7-380b-476e-b255-be186961cedf-kube-api-access-2nmjh\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478656    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62085ba7-380b-476e-b255-be186961cedf-metrics-client-ca\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:23:10.478690    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/62085ba7-380b-476e-b255-be186961cedf-node-exporter-accelerators-collector-config\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/62085ba7-380b-476e-b255-be186961cedf-sys\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/62085ba7-380b-476e-b255-be186961cedf-root\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:23:10.478769    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-tls podName:62085ba7-380b-476e-b255-be186961cedf nodeName:}" failed. No retries permitted until 2026-09-06 10:23:10.978747207 +0000 UTC m=+82.251388705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-tls") pod "node-exporter-2x2n2" (UID: "62085ba7-380b-476e-b255-be186961cedf") : secret "node-exporter-tls" not found
Sep 06 10:23:10.478878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478697    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/62085ba7-380b-476e-b255-be186961cedf-node-exporter-wtmp\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.479392 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.478914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/62085ba7-380b-476e-b255-be186961cedf-node-exporter-textfile\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.479392 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.479301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/62085ba7-380b-476e-b255-be186961cedf-node-exporter-accelerators-collector-config\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.479392 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.479301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62085ba7-380b-476e-b255-be186961cedf-metrics-client-ca\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.480918 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.480898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.487795 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.487767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmjh\" (UniqueName: \"kubernetes.io/projected/62085ba7-380b-476e-b255-be186961cedf-kube-api-access-2nmjh\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.982635 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:10.982589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-tls\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:10.982826 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:23:10.982710    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 06 10:23:10.982826 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:23:10.982782    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-tls podName:62085ba7-380b-476e-b255-be186961cedf nodeName:}" failed. No retries permitted until 2026-09-06 10:23:11.982768308 +0000 UTC m=+83.255409789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-tls") pod "node-exporter-2x2n2" (UID: "62085ba7-380b-476e-b255-be186961cedf") : secret "node-exporter-tls" not found
Sep 06 10:23:11.991665 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:11.991624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-tls\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:11.993908 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:11.993886    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/62085ba7-380b-476e-b255-be186961cedf-node-exporter-tls\") pod \"node-exporter-2x2n2\" (UID: \"62085ba7-380b-476e-b255-be186961cedf\") " pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:12.132193 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:12.132156    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2x2n2"
Sep 06 10:23:12.144103 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:23:12.144075    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62085ba7_380b_476e_b255_be186961cedf.slice/crio-e1165b1d2fa17637c3e821cfeee2af0d709bdb33ec2414b014505ffbc6ca3a24 WatchSource:0}: Error finding container e1165b1d2fa17637c3e821cfeee2af0d709bdb33ec2414b014505ffbc6ca3a24: Status 404 returned error can't find the container with id e1165b1d2fa17637c3e821cfeee2af0d709bdb33ec2414b014505ffbc6ca3a24
Sep 06 10:23:12.731090 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:12.731052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2x2n2" event={"ID":"62085ba7-380b-476e-b255-be186961cedf","Type":"ContainerStarted","Data":"e1165b1d2fa17637c3e821cfeee2af0d709bdb33ec2414b014505ffbc6ca3a24"}
Sep 06 10:23:13.736126 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:13.736089    2583 generic.go:356] "Generic (PLEG): container finished" podID="62085ba7-380b-476e-b255-be186961cedf" containerID="732497eb974c44fd0f482af132b808a0baff7fd164979795f324c08c998857d8" exitCode=0
Sep 06 10:23:13.736533 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:13.736175    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2x2n2" event={"ID":"62085ba7-380b-476e-b255-be186961cedf","Type":"ContainerDied","Data":"732497eb974c44fd0f482af132b808a0baff7fd164979795f324c08c998857d8"}
Sep 06 10:23:14.741973 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:14.741933    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2x2n2" event={"ID":"62085ba7-380b-476e-b255-be186961cedf","Type":"ContainerStarted","Data":"47281c05d54d3dfc8d67ed1f27387ac9a196121c7a07fb6d0cd0ba90a246810e"}
Sep 06 10:23:14.742470 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:14.741988    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2x2n2" event={"ID":"62085ba7-380b-476e-b255-be186961cedf","Type":"ContainerStarted","Data":"7cc68dadc9419c7dc415c082d404c7b0c8b3c06e605807044eca84d11ad44753"}
Sep 06 10:23:14.763630 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:14.763562    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2x2n2" podStartSLOduration=3.673768568 podStartE2EDuration="4.763544531s" totalImagesPullingTime="1.027946033s" totalInitContainerRuntime="61.82993ms" isStatefulPod=true podCreationTimestamp="2026-09-06 10:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:23:14.762134314 +0000 UTC m=+86.034775816" watchObservedRunningTime="2026-09-06 10:23:14.763544531 +0000 UTC m=+86.036186034"
Sep 06 10:23:17.656063 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:17.656031    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:23:22.676618 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:22.676576    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4" podUID="13a64550-4561-48a1-8348-a08076259021" containerName="registry" containerID="cri-o://9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0" gracePeriod=30
Sep 06 10:23:22.909416 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:22.909389    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:23:23.075955 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.075854    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-bound-sa-token\") pod \"13a64550-4561-48a1-8348-a08076259021\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") "
Sep 06 10:23:23.075955 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.075909    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") pod \"13a64550-4561-48a1-8348-a08076259021\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") "
Sep 06 10:23:23.075955 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.075933    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8m27\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-kube-api-access-v8m27\") pod \"13a64550-4561-48a1-8348-a08076259021\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") "
Sep 06 10:23:23.076226 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.075963    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-installation-pull-secrets\") pod \"13a64550-4561-48a1-8348-a08076259021\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") "
Sep 06 10:23:23.076226 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.075985    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-image-registry-private-configuration\") pod \"13a64550-4561-48a1-8348-a08076259021\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") "
Sep 06 10:23:23.076226 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.076011    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-registry-certificates\") pod \"13a64550-4561-48a1-8348-a08076259021\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") "
Sep 06 10:23:23.076226 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.076034    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-trusted-ca\") pod \"13a64550-4561-48a1-8348-a08076259021\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") "
Sep 06 10:23:23.076226 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.076077    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a64550-4561-48a1-8348-a08076259021-ca-trust-extracted\") pod \"13a64550-4561-48a1-8348-a08076259021\" (UID: \"13a64550-4561-48a1-8348-a08076259021\") "
Sep 06 10:23:23.076776 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.076676    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "13a64550-4561-48a1-8348-a08076259021" (UID: "13a64550-4561-48a1-8348-a08076259021"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:23:23.076969 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.076916    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "13a64550-4561-48a1-8348-a08076259021" (UID: "13a64550-4561-48a1-8348-a08076259021"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:23:23.078753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.078709    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "13a64550-4561-48a1-8348-a08076259021" (UID: "13a64550-4561-48a1-8348-a08076259021"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:23:23.078872 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.078812    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "13a64550-4561-48a1-8348-a08076259021" (UID: "13a64550-4561-48a1-8348-a08076259021"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:23:23.078872 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.078831    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "13a64550-4561-48a1-8348-a08076259021" (UID: "13a64550-4561-48a1-8348-a08076259021"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:23:23.078972 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.078871    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-kube-api-access-v8m27" (OuterVolumeSpecName: "kube-api-access-v8m27") pod "13a64550-4561-48a1-8348-a08076259021" (UID: "13a64550-4561-48a1-8348-a08076259021"). InnerVolumeSpecName "kube-api-access-v8m27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:23:23.079047 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.079025    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "13a64550-4561-48a1-8348-a08076259021" (UID: "13a64550-4561-48a1-8348-a08076259021"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:23:23.084840 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.084814    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13a64550-4561-48a1-8348-a08076259021-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "13a64550-4561-48a1-8348-a08076259021" (UID: "13a64550-4561-48a1-8348-a08076259021"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:23:23.177190 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.177140    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-bound-sa-token\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:23:23.177190 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.177183    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-registry-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:23:23.177190 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.177194    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8m27\" (UniqueName: \"kubernetes.io/projected/13a64550-4561-48a1-8348-a08076259021-kube-api-access-v8m27\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:23:23.177190 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.177202    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-installation-pull-secrets\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:23:23.177472 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.177212    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/13a64550-4561-48a1-8348-a08076259021-image-registry-private-configuration\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:23:23.177472 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.177225    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-registry-certificates\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:23:23.177472 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.177236    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/13a64550-4561-48a1-8348-a08076259021-trusted-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:23:23.177472 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.177245    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/13a64550-4561-48a1-8348-a08076259021-ca-trust-extracted\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:23:23.774317 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.774283    2583 generic.go:356] "Generic (PLEG): container finished" podID="13a64550-4561-48a1-8348-a08076259021" containerID="9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0" exitCode=0
Sep 06 10:23:23.774748 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.774327    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4"
Sep 06 10:23:23.774748 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.774365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4" event={"ID":"13a64550-4561-48a1-8348-a08076259021","Type":"ContainerDied","Data":"9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0"}
Sep 06 10:23:23.774748 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.774411    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c867cc788-k5bm4" event={"ID":"13a64550-4561-48a1-8348-a08076259021","Type":"ContainerDied","Data":"b43d7bf432cee8bbfdaa0ed920803c8bfdef2e022926c89ad6913615672cb295"}
Sep 06 10:23:23.774748 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.774442    2583 scope.go:117] "RemoveContainer" containerID="9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0"
Sep 06 10:23:23.782590 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.782573    2583 scope.go:117] "RemoveContainer" containerID="9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0"
Sep 06 10:23:23.782887 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:23:23.782865    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0\": container with ID starting with 9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0 not found: ID does not exist" containerID="9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0"
Sep 06 10:23:23.782926 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.782896    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0"} err="failed to get container status \"9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0\": rpc error: code = NotFound desc = could not find container \"9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0\": container with ID starting with 9d55fb195f1e527d7b9fb7b593d71b3eeac238c388237044c4cd8f83b53a1bd0 not found: ID does not exist"
Sep 06 10:23:23.788856 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.788828    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c867cc788-k5bm4"]
Sep 06 10:23:23.793691 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:23.793670    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c867cc788-k5bm4"]
Sep 06 10:23:25.238631 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:25.238600    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13a64550-4561-48a1-8348-a08076259021" path="/var/lib/kubelet/pods/13a64550-4561-48a1-8348-a08076259021/volumes"
Sep 06 10:23:36.708151 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:23:36.708119    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2j5x7"
Sep 06 10:25:49.321975 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.321938    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"]
Sep 06 10:25:49.322429 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.322183    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13a64550-4561-48a1-8348-a08076259021" containerName="registry"
Sep 06 10:25:49.322429 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.322195    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="13a64550-4561-48a1-8348-a08076259021" containerName="registry"
Sep 06 10:25:49.322429 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.322247    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="13a64550-4561-48a1-8348-a08076259021" containerName="registry"
Sep 06 10:25:49.324766 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.324750    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"
Sep 06 10:25:49.326953 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.326924    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 06 10:25:49.327056 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.326979    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 06 10:25:49.327378 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.327361    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-zg58p\""
Sep 06 10:25:49.334284 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.334258    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"]
Sep 06 10:25:49.502762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.502724    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvhq\" (UniqueName: \"kubernetes.io/projected/e89f0c1c-04f3-4f2a-8676-7b23ccca2c97-kube-api-access-swvhq\") pod \"openshift-lws-operator-fd8ccff4c-lv6kk\" (UID: \"e89f0c1c-04f3-4f2a-8676-7b23ccca2c97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"
Sep 06 10:25:49.502935 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.502832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e89f0c1c-04f3-4f2a-8676-7b23ccca2c97-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lv6kk\" (UID: \"e89f0c1c-04f3-4f2a-8676-7b23ccca2c97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"
Sep 06 10:25:49.603170 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.603134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvhq\" (UniqueName: \"kubernetes.io/projected/e89f0c1c-04f3-4f2a-8676-7b23ccca2c97-kube-api-access-swvhq\") pod \"openshift-lws-operator-fd8ccff4c-lv6kk\" (UID: \"e89f0c1c-04f3-4f2a-8676-7b23ccca2c97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"
Sep 06 10:25:49.603385 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.603191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e89f0c1c-04f3-4f2a-8676-7b23ccca2c97-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lv6kk\" (UID: \"e89f0c1c-04f3-4f2a-8676-7b23ccca2c97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"
Sep 06 10:25:49.603535 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.603517    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e89f0c1c-04f3-4f2a-8676-7b23ccca2c97-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lv6kk\" (UID: \"e89f0c1c-04f3-4f2a-8676-7b23ccca2c97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"
Sep 06 10:25:49.610587 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.610552    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvhq\" (UniqueName: \"kubernetes.io/projected/e89f0c1c-04f3-4f2a-8676-7b23ccca2c97-kube-api-access-swvhq\") pod \"openshift-lws-operator-fd8ccff4c-lv6kk\" (UID: \"e89f0c1c-04f3-4f2a-8676-7b23ccca2c97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"
Sep 06 10:25:49.634125 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.634087    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"
Sep 06 10:25:49.903368 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:49.903279    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk"]
Sep 06 10:25:49.906858 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:25:49.906831    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode89f0c1c_04f3_4f2a_8676_7b23ccca2c97.slice/crio-4469c839ea82cfd00c27d2a160ad30f746a90cfd1e25947ba1d70382dea1a44f WatchSource:0}: Error finding container 4469c839ea82cfd00c27d2a160ad30f746a90cfd1e25947ba1d70382dea1a44f: Status 404 returned error can't find the container with id 4469c839ea82cfd00c27d2a160ad30f746a90cfd1e25947ba1d70382dea1a44f
Sep 06 10:25:50.254451 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:50.254365    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk" event={"ID":"e89f0c1c-04f3-4f2a-8676-7b23ccca2c97","Type":"ContainerStarted","Data":"4469c839ea82cfd00c27d2a160ad30f746a90cfd1e25947ba1d70382dea1a44f"}
Sep 06 10:25:51.702505 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.702469    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4smjm"]
Sep 06 10:25:51.713928 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.713897    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4smjm"
Sep 06 10:25:51.715886 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.715858    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 06 10:25:51.716041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.715891    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-5wbdq\""
Sep 06 10:25:51.716041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.715935    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4smjm"]
Sep 06 10:25:51.716041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.715943    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 06 10:25:51.718426 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.718404    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbnz\" (UniqueName: \"kubernetes.io/projected/609d2b4c-10af-47eb-8524-26fe255099cc-kube-api-access-7lbnz\") pod \"cert-manager-55b4d954d5-4smjm\" (UID: \"609d2b4c-10af-47eb-8524-26fe255099cc\") " pod="cert-manager/cert-manager-55b4d954d5-4smjm"
Sep 06 10:25:51.718555 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.718448    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/609d2b4c-10af-47eb-8524-26fe255099cc-bound-sa-token\") pod \"cert-manager-55b4d954d5-4smjm\" (UID: \"609d2b4c-10af-47eb-8524-26fe255099cc\") " pod="cert-manager/cert-manager-55b4d954d5-4smjm"
Sep 06 10:25:51.819450 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.819415    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/609d2b4c-10af-47eb-8524-26fe255099cc-bound-sa-token\") pod \"cert-manager-55b4d954d5-4smjm\" (UID: \"609d2b4c-10af-47eb-8524-26fe255099cc\") " pod="cert-manager/cert-manager-55b4d954d5-4smjm"
Sep 06 10:25:51.819646 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.819477    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbnz\" (UniqueName: \"kubernetes.io/projected/609d2b4c-10af-47eb-8524-26fe255099cc-kube-api-access-7lbnz\") pod \"cert-manager-55b4d954d5-4smjm\" (UID: \"609d2b4c-10af-47eb-8524-26fe255099cc\") " pod="cert-manager/cert-manager-55b4d954d5-4smjm"
Sep 06 10:25:51.828581 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.828549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/609d2b4c-10af-47eb-8524-26fe255099cc-bound-sa-token\") pod \"cert-manager-55b4d954d5-4smjm\" (UID: \"609d2b4c-10af-47eb-8524-26fe255099cc\") " pod="cert-manager/cert-manager-55b4d954d5-4smjm"
Sep 06 10:25:51.828745 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:51.828654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbnz\" (UniqueName: \"kubernetes.io/projected/609d2b4c-10af-47eb-8524-26fe255099cc-kube-api-access-7lbnz\") pod \"cert-manager-55b4d954d5-4smjm\" (UID: \"609d2b4c-10af-47eb-8524-26fe255099cc\") " pod="cert-manager/cert-manager-55b4d954d5-4smjm"
Sep 06 10:25:52.025601 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:52.025516    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-4smjm"
Sep 06 10:25:52.443468 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:52.443434    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-4smjm"]
Sep 06 10:25:52.446634 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:25:52.446605    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod609d2b4c_10af_47eb_8524_26fe255099cc.slice/crio-11c761de541416d11b71f8567e05a70a330719ce0487c621043f4faa21bf21b7 WatchSource:0}: Error finding container 11c761de541416d11b71f8567e05a70a330719ce0487c621043f4faa21bf21b7: Status 404 returned error can't find the container with id 11c761de541416d11b71f8567e05a70a330719ce0487c621043f4faa21bf21b7
Sep 06 10:25:53.266204 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:53.266162    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk" event={"ID":"e89f0c1c-04f3-4f2a-8676-7b23ccca2c97","Type":"ContainerStarted","Data":"71a0ef2852a403b19e560ca9d4418cf312abbf5c4f8bf607cca35b79e3b9af61"}
Sep 06 10:25:53.267413 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:53.267391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4smjm" event={"ID":"609d2b4c-10af-47eb-8524-26fe255099cc","Type":"ContainerStarted","Data":"11c761de541416d11b71f8567e05a70a330719ce0487c621043f4faa21bf21b7"}
Sep 06 10:25:53.281605 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:53.281558    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lv6kk" podStartSLOduration=1.952363692 podStartE2EDuration="4.281541114s" totalImagesPullingTime="2.329177422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:25:53.280295598 +0000 UTC m=+244.552937100" watchObservedRunningTime="2026-09-06 10:25:53.281541114 +0000 UTC m=+244.554182676"
Sep 06 10:25:56.281091 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:56.281054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-4smjm" event={"ID":"609d2b4c-10af-47eb-8524-26fe255099cc","Type":"ContainerStarted","Data":"d8212d48d1cf7f61b4403ce345536fd974621dab6e6dca0e5c0fc84a3f99f34e"}
Sep 06 10:25:56.295440 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:25:56.295385    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-4smjm" podStartSLOduration=1.940326573 podStartE2EDuration="5.295368996s" totalImagesPullingTime="3.355042423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:25:56.294755284 +0000 UTC m=+247.567396787" watchObservedRunningTime="2026-09-06 10:25:56.295368996 +0000 UTC m=+247.568010498"
Sep 06 10:26:08.447657 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.447620    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"]
Sep 06 10:26:08.451142 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.451117    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.453539 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.453510    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 06 10:26:08.453539 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.453525    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-w5pz8\""
Sep 06 10:26:08.453793 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.453559    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 06 10:26:08.453793 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.453786    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 06 10:26:08.454015 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.454000    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 06 10:26:08.465221 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.465186    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"]
Sep 06 10:26:08.535804 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.535763    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d9e9556-7d42-4c39-b689-4e5d9dd51711-manifests\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.535979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.535813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmh5t\" (UniqueName: \"kubernetes.io/projected/0d9e9556-7d42-4c39-b689-4e5d9dd51711-kube-api-access-rmh5t\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.535979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.535905    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d9e9556-7d42-4c39-b689-4e5d9dd51711-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.535979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.535925    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d9e9556-7d42-4c39-b689-4e5d9dd51711-webhook-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.535979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.535957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d9e9556-7d42-4c39-b689-4e5d9dd51711-tmp\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.637138 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.637080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d9e9556-7d42-4c39-b689-4e5d9dd51711-tmp\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.637138 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.637143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d9e9556-7d42-4c39-b689-4e5d9dd51711-manifests\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.637439 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.637179    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmh5t\" (UniqueName: \"kubernetes.io/projected/0d9e9556-7d42-4c39-b689-4e5d9dd51711-kube-api-access-rmh5t\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.637439 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.637248    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d9e9556-7d42-4c39-b689-4e5d9dd51711-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.637439 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.637270    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d9e9556-7d42-4c39-b689-4e5d9dd51711-webhook-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.637649 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.637624    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0d9e9556-7d42-4c39-b689-4e5d9dd51711-manifests\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.639504 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.639479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d9e9556-7d42-4c39-b689-4e5d9dd51711-tmp\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.639811 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.639791    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0d9e9556-7d42-4c39-b689-4e5d9dd51711-webhook-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.639811 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.639802    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0d9e9556-7d42-4c39-b689-4e5d9dd51711-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.645629 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.645602    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmh5t\" (UniqueName: \"kubernetes.io/projected/0d9e9556-7d42-4c39-b689-4e5d9dd51711-kube-api-access-rmh5t\") pod \"opendatahub-operator-controller-manager-c688cbfb6-9qw6m\" (UID: \"0d9e9556-7d42-4c39-b689-4e5d9dd51711\") " pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:08.762080 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:08.761982    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:09.038706 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:09.038617    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"]
Sep 06 10:26:09.042195 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:26:09.042160    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9e9556_7d42_4c39_b689_4e5d9dd51711.slice/crio-8aac31a68342732981628a713402f5497037f76990c9cbc8bebf34a31a660348 WatchSource:0}: Error finding container 8aac31a68342732981628a713402f5497037f76990c9cbc8bebf34a31a660348: Status 404 returned error can't find the container with id 8aac31a68342732981628a713402f5497037f76990c9cbc8bebf34a31a660348
Sep 06 10:26:09.325976 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:09.325891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m" event={"ID":"0d9e9556-7d42-4c39-b689-4e5d9dd51711","Type":"ContainerStarted","Data":"8aac31a68342732981628a713402f5497037f76990c9cbc8bebf34a31a660348"}
Sep 06 10:26:12.339627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:12.339594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m" event={"ID":"0d9e9556-7d42-4c39-b689-4e5d9dd51711","Type":"ContainerStarted","Data":"6387a0e8c57e8cfc701d68b9e7037ce04b548e2d23f7015efbd9de1bff1b2e54"}
Sep 06 10:26:13.344862 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:13.344828    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d9e9556-7d42-4c39-b689-4e5d9dd51711" containerID="6387a0e8c57e8cfc701d68b9e7037ce04b548e2d23f7015efbd9de1bff1b2e54" exitCode=0
Sep 06 10:26:13.345245 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:13.344892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m" event={"ID":"0d9e9556-7d42-4c39-b689-4e5d9dd51711","Type":"ContainerDied","Data":"6387a0e8c57e8cfc701d68b9e7037ce04b548e2d23f7015efbd9de1bff1b2e54"}
Sep 06 10:26:14.353270 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:14.353235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m" event={"ID":"0d9e9556-7d42-4c39-b689-4e5d9dd51711","Type":"ContainerStarted","Data":"37f6196d000bf292d858745708f993992dc896d75a0ade289963baabe21c3587"}
Sep 06 10:26:14.353753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:14.353480    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:14.374228 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:14.374173    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m" podStartSLOduration=1.834933575 podStartE2EDuration="6.374158377s" totalImagesPullingTime="3.499521055s" totalInitContainerRuntime="1.039703747s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:26:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:26:14.373709458 +0000 UTC m=+265.646350963" watchObservedRunningTime="2026-09-06 10:26:14.374158377 +0000 UTC m=+265.646799880"
Sep 06 10:26:25.358882 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:25.358851    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c688cbfb6-9qw6m"
Sep 06 10:26:27.483948 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.483910    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"]
Sep 06 10:26:27.592202 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.592157    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"]
Sep 06 10:26:27.592395 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.592282    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.604587 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.604561    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 06 10:26:27.605029 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.605011    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2l794\""
Sep 06 10:26:27.605029 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.605019    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 06 10:26:27.605202 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.605050    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 06 10:26:27.605202 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.605151    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 06 10:26:27.682378 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.682312    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2tw\" (UniqueName: \"kubernetes.io/projected/1a1ae590-4d8e-443a-9abd-53ebc09b4c25-kube-api-access-5v2tw\") pod \"kube-auth-proxy-68f5945d9f-jktb4\" (UID: \"1a1ae590-4d8e-443a-9abd-53ebc09b4c25\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.682561 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.682430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1ae590-4d8e-443a-9abd-53ebc09b4c25-tls-certs\") pod \"kube-auth-proxy-68f5945d9f-jktb4\" (UID: \"1a1ae590-4d8e-443a-9abd-53ebc09b4c25\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.682561 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.682468    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae590-4d8e-443a-9abd-53ebc09b4c25-tmp\") pod \"kube-auth-proxy-68f5945d9f-jktb4\" (UID: \"1a1ae590-4d8e-443a-9abd-53ebc09b4c25\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.783238 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.783147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2tw\" (UniqueName: \"kubernetes.io/projected/1a1ae590-4d8e-443a-9abd-53ebc09b4c25-kube-api-access-5v2tw\") pod \"kube-auth-proxy-68f5945d9f-jktb4\" (UID: \"1a1ae590-4d8e-443a-9abd-53ebc09b4c25\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.783238 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.783215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1ae590-4d8e-443a-9abd-53ebc09b4c25-tls-certs\") pod \"kube-auth-proxy-68f5945d9f-jktb4\" (UID: \"1a1ae590-4d8e-443a-9abd-53ebc09b4c25\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.783496 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.783253    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae590-4d8e-443a-9abd-53ebc09b4c25-tmp\") pod \"kube-auth-proxy-68f5945d9f-jktb4\" (UID: \"1a1ae590-4d8e-443a-9abd-53ebc09b4c25\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.785613 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.785581    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a1ae590-4d8e-443a-9abd-53ebc09b4c25-tmp\") pod \"kube-auth-proxy-68f5945d9f-jktb4\" (UID: \"1a1ae590-4d8e-443a-9abd-53ebc09b4c25\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.785782 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.785765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a1ae590-4d8e-443a-9abd-53ebc09b4c25-tls-certs\") pod \"kube-auth-proxy-68f5945d9f-jktb4\" (UID: \"1a1ae590-4d8e-443a-9abd-53ebc09b4c25\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.807321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.807291    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2tw\" (UniqueName: \"kubernetes.io/projected/1a1ae590-4d8e-443a-9abd-53ebc09b4c25-kube-api-access-5v2tw\") pod \"kube-auth-proxy-68f5945d9f-jktb4\" (UID: \"1a1ae590-4d8e-443a-9abd-53ebc09b4c25\") " pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:27.901428 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:27.901392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"
Sep 06 10:26:28.190631 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.190594    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4"]
Sep 06 10:26:28.193991 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:26:28.193966    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1ae590_4d8e_443a_9abd_53ebc09b4c25.slice/crio-c9dc93c92bf52db02949ff3cbf7a7adcbad6a5276537dd9a76078f09618405cd WatchSource:0}: Error finding container c9dc93c92bf52db02949ff3cbf7a7adcbad6a5276537dd9a76078f09618405cd: Status 404 returned error can't find the container with id c9dc93c92bf52db02949ff3cbf7a7adcbad6a5276537dd9a76078f09618405cd
Sep 06 10:26:28.404740 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.404709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4" event={"ID":"1a1ae590-4d8e-443a-9abd-53ebc09b4c25","Type":"ContainerStarted","Data":"c9dc93c92bf52db02949ff3cbf7a7adcbad6a5276537dd9a76078f09618405cd"}
Sep 06 10:26:28.861137 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.861089    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"]
Sep 06 10:26:28.874832 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.874797    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"]
Sep 06 10:26:28.875003 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.874903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:28.878064 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.878037    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 06 10:26:28.878391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.878370    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 06 10:26:28.878601 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.878571    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lqjmz\""
Sep 06 10:26:28.878697 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.878608    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 06 10:26:28.878751 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.878578    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 06 10:26:28.992154 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.992114    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:28.992374 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.992226    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:28.992374 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.992283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/488bca64-c134-434f-b3c8-f061da990e0f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:28.992374 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.992353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:28.992559 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.992394    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:28.992559 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.992463    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:28.992559 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:28.992500    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzwp\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-kube-api-access-slzwp\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.093328 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.093286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.093527 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.093365    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.093527 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.093409    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/488bca64-c134-434f-b3c8-f061da990e0f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.093527 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.093461    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.093527 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.093498    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.093742 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.093536    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.093742 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.093565    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slzwp\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-kube-api-access-slzwp\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.094085 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.094059    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.096598 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.096510    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.096862 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.096832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/488bca64-c134-434f-b3c8-f061da990e0f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.097219 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.097191    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.097933 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.097908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.101925 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.101902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.102378 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.102355    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzwp\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-kube-api-access-slzwp\") pod \"istiod-openshift-gateway-7b7995d5cd-kkdpb\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.189099 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.188826    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:29.507006 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:29.506935    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"]
Sep 06 10:26:29.510475 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:26:29.510423    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488bca64_c134_434f_b3c8_f061da990e0f.slice/crio-4d755d1d3b7bb1858f70a6afb5608778e23991ac37dbf0c03fa5b45aa0cbe2f3 WatchSource:0}: Error finding container 4d755d1d3b7bb1858f70a6afb5608778e23991ac37dbf0c03fa5b45aa0cbe2f3: Status 404 returned error can't find the container with id 4d755d1d3b7bb1858f70a6afb5608778e23991ac37dbf0c03fa5b45aa0cbe2f3
Sep 06 10:26:30.417784 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:30.417750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb" event={"ID":"488bca64-c134-434f-b3c8-f061da990e0f","Type":"ContainerStarted","Data":"4d755d1d3b7bb1858f70a6afb5608778e23991ac37dbf0c03fa5b45aa0cbe2f3"}
Sep 06 10:26:31.424189 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:31.424137    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4" event={"ID":"1a1ae590-4d8e-443a-9abd-53ebc09b4c25","Type":"ContainerStarted","Data":"3a7f9520a7e8a1273f9e581870b6104b9fb8cacda33aaa01e417c39afc1e7370"}
Sep 06 10:26:31.469048 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:31.468977    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68f5945d9f-jktb4" podStartSLOduration=1.6400576949999999 podStartE2EDuration="4.468957858s" totalImagesPullingTime="2.828900163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:26:31.467946955 +0000 UTC m=+282.740588465" watchObservedRunningTime="2026-09-06 10:26:31.468957858 +0000 UTC m=+282.741599361"
Sep 06 10:26:32.769170 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:32.769113    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 10:26:32.769433 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:32.769214    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 10:26:33.435108 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:33.435065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb" event={"ID":"488bca64-c134-434f-b3c8-f061da990e0f","Type":"ContainerStarted","Data":"29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835"}
Sep 06 10:26:33.435284 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:33.435232    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:33.436852 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:33.436824    2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-kkdpb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 06 10:26:33.436967 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:33.436874    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb" podUID="488bca64-c134-434f-b3c8-f061da990e0f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 06 10:26:33.456210 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:33.456155    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb" podStartSLOduration=2.200068851 podStartE2EDuration="5.456139239s" totalImagesPullingTime="3.256070388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:26:33.454615508 +0000 UTC m=+284.727257011" watchObservedRunningTime="2026-09-06 10:26:33.456139239 +0000 UTC m=+284.728780741"
Sep 06 10:26:34.439622 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:34.439590    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:38.207605 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.207563    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8vnx9"]
Sep 06 10:26:38.210628 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.210609    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:38.212510 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.212489    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 06 10:26:38.212611 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.212517    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dztmf\""
Sep 06 10:26:38.219329 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.219306    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8vnx9"]
Sep 06 10:26:38.273256 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.273214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdfzx\" (UniqueName: \"kubernetes.io/projected/46181b28-9d0e-4374-96a9-7df450e6333b-kube-api-access-rdfzx\") pod \"odh-model-controller-7998c67dc4-8vnx9\" (UID: \"46181b28-9d0e-4374-96a9-7df450e6333b\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:38.273449 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.273273    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46181b28-9d0e-4374-96a9-7df450e6333b-cert\") pod \"odh-model-controller-7998c67dc4-8vnx9\" (UID: \"46181b28-9d0e-4374-96a9-7df450e6333b\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:38.373650 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.373621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46181b28-9d0e-4374-96a9-7df450e6333b-cert\") pod \"odh-model-controller-7998c67dc4-8vnx9\" (UID: \"46181b28-9d0e-4374-96a9-7df450e6333b\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:38.373789 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.373707    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdfzx\" (UniqueName: \"kubernetes.io/projected/46181b28-9d0e-4374-96a9-7df450e6333b-kube-api-access-rdfzx\") pod \"odh-model-controller-7998c67dc4-8vnx9\" (UID: \"46181b28-9d0e-4374-96a9-7df450e6333b\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:38.373789 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:26:38.373759    2583 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 06 10:26:38.373850 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:26:38.373841    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46181b28-9d0e-4374-96a9-7df450e6333b-cert podName:46181b28-9d0e-4374-96a9-7df450e6333b nodeName:}" failed. No retries permitted until 2026-09-06 10:26:38.873822027 +0000 UTC m=+290.146463508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/46181b28-9d0e-4374-96a9-7df450e6333b-cert") pod "odh-model-controller-7998c67dc4-8vnx9" (UID: "46181b28-9d0e-4374-96a9-7df450e6333b") : secret "odh-model-controller-webhook-cert" not found
Sep 06 10:26:38.382581 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.382556    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdfzx\" (UniqueName: \"kubernetes.io/projected/46181b28-9d0e-4374-96a9-7df450e6333b-kube-api-access-rdfzx\") pod \"odh-model-controller-7998c67dc4-8vnx9\" (UID: \"46181b28-9d0e-4374-96a9-7df450e6333b\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:38.796355 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.796301    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nxqbr"]
Sep 06 10:26:38.799493 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.799475    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:38.801357 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.801314    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 06 10:26:38.801480 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.801439    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fpsdp\""
Sep 06 10:26:38.808185 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.808156    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nxqbr"]
Sep 06 10:26:38.878182 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.878138    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474c1520-fc8c-4049-86f3-4d2331b70da1-tls-certs\") pod \"model-serving-api-54d89bd5fc-nxqbr\" (UID: \"474c1520-fc8c-4049-86f3-4d2331b70da1\") " pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:38.878182 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.878186    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4kfx\" (UniqueName: \"kubernetes.io/projected/474c1520-fc8c-4049-86f3-4d2331b70da1-kube-api-access-z4kfx\") pod \"model-serving-api-54d89bd5fc-nxqbr\" (UID: \"474c1520-fc8c-4049-86f3-4d2331b70da1\") " pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:38.878415 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.878218    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46181b28-9d0e-4374-96a9-7df450e6333b-cert\") pod \"odh-model-controller-7998c67dc4-8vnx9\" (UID: \"46181b28-9d0e-4374-96a9-7df450e6333b\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:38.880726 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.880702    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46181b28-9d0e-4374-96a9-7df450e6333b-cert\") pod \"odh-model-controller-7998c67dc4-8vnx9\" (UID: \"46181b28-9d0e-4374-96a9-7df450e6333b\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:38.978955 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.978916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474c1520-fc8c-4049-86f3-4d2331b70da1-tls-certs\") pod \"model-serving-api-54d89bd5fc-nxqbr\" (UID: \"474c1520-fc8c-4049-86f3-4d2331b70da1\") " pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:38.978955 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.978961    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4kfx\" (UniqueName: \"kubernetes.io/projected/474c1520-fc8c-4049-86f3-4d2331b70da1-kube-api-access-z4kfx\") pod \"model-serving-api-54d89bd5fc-nxqbr\" (UID: \"474c1520-fc8c-4049-86f3-4d2331b70da1\") " pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:38.979158 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:26:38.979075    2583 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 06 10:26:38.979193 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:26:38.979167    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/474c1520-fc8c-4049-86f3-4d2331b70da1-tls-certs podName:474c1520-fc8c-4049-86f3-4d2331b70da1 nodeName:}" failed. No retries permitted until 2026-09-06 10:26:39.479150342 +0000 UTC m=+290.751791823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/474c1520-fc8c-4049-86f3-4d2331b70da1-tls-certs") pod "model-serving-api-54d89bd5fc-nxqbr" (UID: "474c1520-fc8c-4049-86f3-4d2331b70da1") : secret "model-serving-api-tls" not found
Sep 06 10:26:38.986997 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:38.986968    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4kfx\" (UniqueName: \"kubernetes.io/projected/474c1520-fc8c-4049-86f3-4d2331b70da1-kube-api-access-z4kfx\") pod \"model-serving-api-54d89bd5fc-nxqbr\" (UID: \"474c1520-fc8c-4049-86f3-4d2331b70da1\") " pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:39.121368 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:39.121308    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:39.389719 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:39.389639    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8vnx9"]
Sep 06 10:26:39.395378 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:26:39.393474    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46181b28_9d0e_4374_96a9_7df450e6333b.slice/crio-f00c6bdfd3239f913e407ba7d018011f7986085bb0bc18f14186d17f31e56edb WatchSource:0}: Error finding container f00c6bdfd3239f913e407ba7d018011f7986085bb0bc18f14186d17f31e56edb: Status 404 returned error can't find the container with id f00c6bdfd3239f913e407ba7d018011f7986085bb0bc18f14186d17f31e56edb
Sep 06 10:26:39.459943 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:39.459895    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9" event={"ID":"46181b28-9d0e-4374-96a9-7df450e6333b","Type":"ContainerStarted","Data":"f00c6bdfd3239f913e407ba7d018011f7986085bb0bc18f14186d17f31e56edb"}
Sep 06 10:26:39.483667 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:39.483637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474c1520-fc8c-4049-86f3-4d2331b70da1-tls-certs\") pod \"model-serving-api-54d89bd5fc-nxqbr\" (UID: \"474c1520-fc8c-4049-86f3-4d2331b70da1\") " pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:39.486133 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:39.486100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474c1520-fc8c-4049-86f3-4d2331b70da1-tls-certs\") pod \"model-serving-api-54d89bd5fc-nxqbr\" (UID: \"474c1520-fc8c-4049-86f3-4d2331b70da1\") " pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:39.711115 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:39.711008    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:39.981922 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:39.981825    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-nxqbr"]
Sep 06 10:26:39.985133 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:26:39.985095    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474c1520_fc8c_4049_86f3_4d2331b70da1.slice/crio-2f934ecadc46caea8bca6f8ed20c7de9031a2afc52edf027d052c312e9e7e922 WatchSource:0}: Error finding container 2f934ecadc46caea8bca6f8ed20c7de9031a2afc52edf027d052c312e9e7e922: Status 404 returned error can't find the container with id 2f934ecadc46caea8bca6f8ed20c7de9031a2afc52edf027d052c312e9e7e922
Sep 06 10:26:40.465708 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:40.465667    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr" event={"ID":"474c1520-fc8c-4049-86f3-4d2331b70da1","Type":"ContainerStarted","Data":"2f934ecadc46caea8bca6f8ed20c7de9031a2afc52edf027d052c312e9e7e922"}
Sep 06 10:26:43.484263 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:43.484225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9" event={"ID":"46181b28-9d0e-4374-96a9-7df450e6333b","Type":"ContainerStarted","Data":"e6d3ac473f94fbc6fdf1c02eb2172905a14422929c1170f964da26f9796be2a9"}
Sep 06 10:26:43.484743 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:43.484362    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:43.485650 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:43.485630    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr" event={"ID":"474c1520-fc8c-4049-86f3-4d2331b70da1","Type":"ContainerStarted","Data":"7b95badbb72f3092a5de754034097e4a402eaa0819618e4698a0646dc4080103"}
Sep 06 10:26:43.485758 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:43.485748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:26:43.500942 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:43.500889    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9" podStartSLOduration=2.478794358 podStartE2EDuration="5.500873321s" totalImagesPullingTime="3.022078963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:26:43.499665088 +0000 UTC m=+294.772306590" watchObservedRunningTime="2026-09-06 10:26:43.500873321 +0000 UTC m=+294.773514823"
Sep 06 10:26:44.525965 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.525906    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr" podStartSLOduration=4.093699649 podStartE2EDuration="6.52588413s" totalImagesPullingTime="2.432184481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:26:43.516498835 +0000 UTC m=+294.789140344" watchObservedRunningTime="2026-09-06 10:26:44.52588413 +0000 UTC m=+295.798525635"
Sep 06 10:26:44.527550 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.527521    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"]
Sep 06 10:26:44.530790 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.530768    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.543135 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.543103    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"]
Sep 06 10:26:44.624408 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.624368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.624582 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.624421    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.624582 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.624458    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.624582 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.624490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.624582 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.624517    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.624714 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.624596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvsv\" (UniqueName: \"kubernetes.io/projected/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-kube-api-access-6bvsv\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.624748 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.624714    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.725128 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.725093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.725128 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.725134    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.725354 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.725240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvsv\" (UniqueName: \"kubernetes.io/projected/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-kube-api-access-6bvsv\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.725354 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.725282    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.725354 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.725308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.725518 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.725357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.725518 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.725388    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.726263 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.726205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.727830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.727807    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.727830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.727819    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.727969 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.727907    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.727969 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.727926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.735026 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.735001    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.735912 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.735890    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvsv\" (UniqueName: \"kubernetes.io/projected/b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8-kube-api-access-6bvsv\") pod \"istiod-openshift-gateway-5677ccb49b-nbhc5\" (UID: \"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:44.839045 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:44.839009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:45.119510 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:45.119436    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"]
Sep 06 10:26:45.122613 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:26:45.122588    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52301ae_87ac_4c12_9b4e_9f8c4d3b90f8.slice/crio-77632c12dc3ed37cfaf3f2454f5778095585c1053445d79a6c4c703a5b66e0d6 WatchSource:0}: Error finding container 77632c12dc3ed37cfaf3f2454f5778095585c1053445d79a6c4c703a5b66e0d6: Status 404 returned error can't find the container with id 77632c12dc3ed37cfaf3f2454f5778095585c1053445d79a6c4c703a5b66e0d6
Sep 06 10:26:45.125607 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:45.125575    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 10:26:45.125718 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:45.125646    2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 06 10:26:45.496160 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:45.496070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5" event={"ID":"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8","Type":"ContainerStarted","Data":"05c8764ecd3fa4dff30462ab9895b025bd4bf95871d7943dd8de05f514799cdf"}
Sep 06 10:26:45.496160 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:45.496132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5" event={"ID":"b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8","Type":"ContainerStarted","Data":"77632c12dc3ed37cfaf3f2454f5778095585c1053445d79a6c4c703a5b66e0d6"}
Sep 06 10:26:45.496488 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:45.496216    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:45.521453 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:45.521401    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5" podStartSLOduration=1.521383847 podStartE2EDuration="1.521383847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:26:45.518824289 +0000 UTC m=+296.791465793" watchObservedRunningTime="2026-09-06 10:26:45.521383847 +0000 UTC m=+296.794025351"
Sep 06 10:26:46.502437 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.502405    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-nbhc5"
Sep 06 10:26:46.578867 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.578834    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"]
Sep 06 10:26:46.579048 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.579028    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb" podUID="488bca64-c134-434f-b3c8-f061da990e0f" containerName="discovery" containerID="cri-o://29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835" gracePeriod=30
Sep 06 10:26:46.818840 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.818811    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:46.944127 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.944082    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-dns-cert\") pod \"488bca64-c134-434f-b3c8-f061da990e0f\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") "
Sep 06 10:26:46.944311 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.944139    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-ca-configmap\") pod \"488bca64-c134-434f-b3c8-f061da990e0f\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") "
Sep 06 10:26:46.944311 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.944163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-cacerts\") pod \"488bca64-c134-434f-b3c8-f061da990e0f\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") "
Sep 06 10:26:46.944311 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.944277    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/488bca64-c134-434f-b3c8-f061da990e0f-local-certs\") pod \"488bca64-c134-434f-b3c8-f061da990e0f\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") "
Sep 06 10:26:46.944505 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.944349    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-istio-token\") pod \"488bca64-c134-434f-b3c8-f061da990e0f\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") "
Sep 06 10:26:46.944505 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.944386    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-kubeconfig\") pod \"488bca64-c134-434f-b3c8-f061da990e0f\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") "
Sep 06 10:26:46.944505 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.944412    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slzwp\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-kube-api-access-slzwp\") pod \"488bca64-c134-434f-b3c8-f061da990e0f\" (UID: \"488bca64-c134-434f-b3c8-f061da990e0f\") "
Sep 06 10:26:46.944650 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.944601    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "488bca64-c134-434f-b3c8-f061da990e0f" (UID: "488bca64-c134-434f-b3c8-f061da990e0f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:26:46.944700 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.944680    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-ca-configmap\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:26:46.947152 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.947106    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "488bca64-c134-434f-b3c8-f061da990e0f" (UID: "488bca64-c134-434f-b3c8-f061da990e0f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:26:46.947293 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.947165    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488bca64-c134-434f-b3c8-f061da990e0f-local-certs" (OuterVolumeSpecName: "local-certs") pod "488bca64-c134-434f-b3c8-f061da990e0f" (UID: "488bca64-c134-434f-b3c8-f061da990e0f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:26:46.947293 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.947201    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "488bca64-c134-434f-b3c8-f061da990e0f" (UID: "488bca64-c134-434f-b3c8-f061da990e0f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:26:46.947293 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.947211    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-cacerts" (OuterVolumeSpecName: "cacerts") pod "488bca64-c134-434f-b3c8-f061da990e0f" (UID: "488bca64-c134-434f-b3c8-f061da990e0f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:26:46.947533 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.947328    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-istio-token" (OuterVolumeSpecName: "istio-token") pod "488bca64-c134-434f-b3c8-f061da990e0f" (UID: "488bca64-c134-434f-b3c8-f061da990e0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:26:46.947592 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:46.947574    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-kube-api-access-slzwp" (OuterVolumeSpecName: "kube-api-access-slzwp") pod "488bca64-c134-434f-b3c8-f061da990e0f" (UID: "488bca64-c134-434f-b3c8-f061da990e0f"). InnerVolumeSpecName "kube-api-access-slzwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:26:47.045118 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.045082    2583 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-cacerts\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:26:47.045118 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.045111    2583 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/488bca64-c134-434f-b3c8-f061da990e0f-local-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:26:47.045118 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.045122    2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-istio-token\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:26:47.045369 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.045130    2583 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-kubeconfig\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:26:47.045369 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.045138    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slzwp\" (UniqueName: \"kubernetes.io/projected/488bca64-c134-434f-b3c8-f061da990e0f-kube-api-access-slzwp\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:26:47.045369 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.045146    2583 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/488bca64-c134-434f-b3c8-f061da990e0f-istio-csr-dns-cert\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:26:47.505489 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.505402    2583 generic.go:356] "Generic (PLEG): container finished" podID="488bca64-c134-434f-b3c8-f061da990e0f" containerID="29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835" exitCode=0
Sep 06 10:26:47.505489 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.505451    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"
Sep 06 10:26:47.505908 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.505491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb" event={"ID":"488bca64-c134-434f-b3c8-f061da990e0f","Type":"ContainerDied","Data":"29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835"}
Sep 06 10:26:47.505908 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.505538    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb" event={"ID":"488bca64-c134-434f-b3c8-f061da990e0f","Type":"ContainerDied","Data":"4d755d1d3b7bb1858f70a6afb5608778e23991ac37dbf0c03fa5b45aa0cbe2f3"}
Sep 06 10:26:47.505908 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.505558    2583 scope.go:117] "RemoveContainer" containerID="29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835"
Sep 06 10:26:47.514118 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.514097    2583 scope.go:117] "RemoveContainer" containerID="29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835"
Sep 06 10:26:47.514439 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:26:47.514416    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835\": container with ID starting with 29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835 not found: ID does not exist" containerID="29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835"
Sep 06 10:26:47.514515 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.514450    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835"} err="failed to get container status \"29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835\": rpc error: code = NotFound desc = could not find container \"29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835\": container with ID starting with 29f4bcdbd8b543e14248ab8e52f1f30889ea3d088a47a8e8f9dadbb786c1f835 not found: ID does not exist"
Sep 06 10:26:47.524288 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.524256    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"]
Sep 06 10:26:47.530899 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:47.530864    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-kkdpb"]
Sep 06 10:26:49.165439 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:49.165414    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 06 10:26:49.239921 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:49.239795    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="488bca64-c134-434f-b3c8-f061da990e0f" path="/var/lib/kubelet/pods/488bca64-c134-434f-b3c8-f061da990e0f/volumes"
Sep 06 10:26:54.492389 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:54.492357    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8vnx9"
Sep 06 10:26:54.494666 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:26:54.494647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-nxqbr"
Sep 06 10:27:54.729902 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.729823    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"]
Sep 06 10:27:54.730295 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.730137    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="488bca64-c134-434f-b3c8-f061da990e0f" containerName="discovery"
Sep 06 10:27:54.730295 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.730150    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="488bca64-c134-434f-b3c8-f061da990e0f" containerName="discovery"
Sep 06 10:27:54.730295 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.730202    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="488bca64-c134-434f-b3c8-f061da990e0f" containerName="discovery"
Sep 06 10:27:54.732968 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.732950    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"
Sep 06 10:27:54.735282 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.735262    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 06 10:27:54.735494 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.735479    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5klng\""
Sep 06 10:27:54.736000 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.735987    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 06 10:27:54.743815 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.743777    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"]
Sep 06 10:27:54.869207 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.869166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wwwv\" (UniqueName: \"kubernetes.io/projected/a85b4ebb-9489-43c4-868d-972c5bee7edb-kube-api-access-7wwwv\") pod \"limitador-operator-controller-manager-6c549b6d85-j8cpg\" (UID: \"a85b4ebb-9489-43c4-868d-972c5bee7edb\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"
Sep 06 10:27:54.970218 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.970168    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wwwv\" (UniqueName: \"kubernetes.io/projected/a85b4ebb-9489-43c4-868d-972c5bee7edb-kube-api-access-7wwwv\") pod \"limitador-operator-controller-manager-6c549b6d85-j8cpg\" (UID: \"a85b4ebb-9489-43c4-868d-972c5bee7edb\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"
Sep 06 10:27:54.982105 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:54.982040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wwwv\" (UniqueName: \"kubernetes.io/projected/a85b4ebb-9489-43c4-868d-972c5bee7edb-kube-api-access-7wwwv\") pod \"limitador-operator-controller-manager-6c549b6d85-j8cpg\" (UID: \"a85b4ebb-9489-43c4-868d-972c5bee7edb\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"
Sep 06 10:27:55.048728 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:55.048689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"
Sep 06 10:27:55.323699 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:55.323613    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"]
Sep 06 10:27:55.326912 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:27:55.326881    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85b4ebb_9489_43c4_868d_972c5bee7edb.slice/crio-8fbf4aee3859f96952a673e757dfeade34d7b5fc08166df0301030d88545be7c WatchSource:0}: Error finding container 8fbf4aee3859f96952a673e757dfeade34d7b5fc08166df0301030d88545be7c: Status 404 returned error can't find the container with id 8fbf4aee3859f96952a673e757dfeade34d7b5fc08166df0301030d88545be7c
Sep 06 10:27:55.329149 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:55.329127    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:27:55.786192 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:55.786151    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg" event={"ID":"a85b4ebb-9489-43c4-868d-972c5bee7edb","Type":"ContainerStarted","Data":"8fbf4aee3859f96952a673e757dfeade34d7b5fc08166df0301030d88545be7c"}
Sep 06 10:27:58.801794 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:58.801754    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg" event={"ID":"a85b4ebb-9489-43c4-868d-972c5bee7edb","Type":"ContainerStarted","Data":"7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2"}
Sep 06 10:27:58.802362 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:58.801839    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"
Sep 06 10:27:58.820274 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:58.820073    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg" podStartSLOduration=1.898664776 podStartE2EDuration="4.820052095s" totalImagesPullingTime="2.921387319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:27:58.819493045 +0000 UTC m=+370.092134549" watchObservedRunningTime="2026-09-06 10:27:58.820052095 +0000 UTC m=+370.092693601"
Sep 06 10:27:58.904843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:58.904808    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"]
Sep 06 10:27:58.908009 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:58.907990    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:27:58.911715 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:58.911692    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-frvzt\""
Sep 06 10:27:58.935038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:58.935005    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"]
Sep 06 10:27:59.004254 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:59.004213    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bb5af51-9c05-4ed2-8a4a-57e84207e337-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-jsszw\" (UID: \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:27:59.004479 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:59.004322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66f7l\" (UniqueName: \"kubernetes.io/projected/0bb5af51-9c05-4ed2-8a4a-57e84207e337-kube-api-access-66f7l\") pod \"kuadrant-operator-controller-manager-58fc56757c-jsszw\" (UID: \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:27:59.105127 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:59.105088    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bb5af51-9c05-4ed2-8a4a-57e84207e337-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-jsszw\" (UID: \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:27:59.105321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:59.105174    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66f7l\" (UniqueName: \"kubernetes.io/projected/0bb5af51-9c05-4ed2-8a4a-57e84207e337-kube-api-access-66f7l\") pod \"kuadrant-operator-controller-manager-58fc56757c-jsszw\" (UID: \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:27:59.105523 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:59.105499    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bb5af51-9c05-4ed2-8a4a-57e84207e337-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-jsszw\" (UID: \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:27:59.113673 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:59.113650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66f7l\" (UniqueName: \"kubernetes.io/projected/0bb5af51-9c05-4ed2-8a4a-57e84207e337-kube-api-access-66f7l\") pod \"kuadrant-operator-controller-manager-58fc56757c-jsszw\" (UID: \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:27:59.218063 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:59.218023    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:27:59.492906 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:59.492827    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"]
Sep 06 10:27:59.496657 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:27:59.496630    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb5af51_9c05_4ed2_8a4a_57e84207e337.slice/crio-982b98b5cb46732cfba0b87d5dad6b0da5e2e413e7976b8aa1ffdb3ea9926d55 WatchSource:0}: Error finding container 982b98b5cb46732cfba0b87d5dad6b0da5e2e413e7976b8aa1ffdb3ea9926d55: Status 404 returned error can't find the container with id 982b98b5cb46732cfba0b87d5dad6b0da5e2e413e7976b8aa1ffdb3ea9926d55
Sep 06 10:27:59.807312 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:27:59.807225    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw" event={"ID":"0bb5af51-9c05-4ed2-8a4a-57e84207e337","Type":"ContainerStarted","Data":"982b98b5cb46732cfba0b87d5dad6b0da5e2e413e7976b8aa1ffdb3ea9926d55"}
Sep 06 10:28:03.828500 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:03.828466    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw" event={"ID":"0bb5af51-9c05-4ed2-8a4a-57e84207e337","Type":"ContainerStarted","Data":"48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681"}
Sep 06 10:28:03.828926 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:03.828548    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:28:03.850543 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:03.850491    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw" podStartSLOduration=1.5962255239999998 podStartE2EDuration="5.850471418s" totalImagesPullingTime="4.254245894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:28:03.848610184 +0000 UTC m=+375.121251687" watchObservedRunningTime="2026-09-06 10:28:03.850471418 +0000 UTC m=+375.123112923"
Sep 06 10:28:09.809566 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:09.809534    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"
Sep 06 10:28:14.835005 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:14.834965    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:28:16.632048 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.632016    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"]
Sep 06 10:28:16.635069 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.635051    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:16.656138 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.656110    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"]
Sep 06 10:28:16.755165 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.755101    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdnz\" (UniqueName: \"kubernetes.io/projected/d26a16cc-9336-4cc3-84c7-d98156f62d21-kube-api-access-dpdnz\") pod \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" (UID: \"d26a16cc-9336-4cc3-84c7-d98156f62d21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:16.755165 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.755174    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d26a16cc-9336-4cc3-84c7-d98156f62d21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" (UID: \"d26a16cc-9336-4cc3-84c7-d98156f62d21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:16.856508 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.856458    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdnz\" (UniqueName: \"kubernetes.io/projected/d26a16cc-9336-4cc3-84c7-d98156f62d21-kube-api-access-dpdnz\") pod \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" (UID: \"d26a16cc-9336-4cc3-84c7-d98156f62d21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:16.856686 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.856528    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d26a16cc-9336-4cc3-84c7-d98156f62d21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" (UID: \"d26a16cc-9336-4cc3-84c7-d98156f62d21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:16.856886 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.856864    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d26a16cc-9336-4cc3-84c7-d98156f62d21-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" (UID: \"d26a16cc-9336-4cc3-84c7-d98156f62d21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:16.864541 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.864511    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdnz\" (UniqueName: \"kubernetes.io/projected/d26a16cc-9336-4cc3-84c7-d98156f62d21-kube-api-access-dpdnz\") pod \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" (UID: \"d26a16cc-9336-4cc3-84c7-d98156f62d21\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:16.945264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:16.945155    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:17.053142 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.053109    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"]
Sep 06 10:28:17.053328 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.053291    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw" podUID="0bb5af51-9c05-4ed2-8a4a-57e84207e337" containerName="manager" containerID="cri-o://48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681" gracePeriod=2
Sep 06 10:28:17.070401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.070371    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"]
Sep 06 10:28:17.092747 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.092709    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"]
Sep 06 10:28:17.102016 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.101985    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"]
Sep 06 10:28:17.102453 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.102437    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb5af51-9c05-4ed2-8a4a-57e84207e337" containerName="manager"
Sep 06 10:28:17.102505 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.102457    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb5af51-9c05-4ed2-8a4a-57e84207e337" containerName="manager"
Sep 06 10:28:17.102553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.102542    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb5af51-9c05-4ed2-8a4a-57e84207e337" containerName="manager"
Sep 06 10:28:17.105653 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.105624    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:17.107179 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.107151    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"]
Sep 06 10:28:17.138764 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.138737    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"]
Sep 06 10:28:17.141834 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.141813    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:17.153900 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.153869    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"]
Sep 06 10:28:17.160582 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:17.160543    2583 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0bb5af51-9c05-4ed2-8a4a-57e84207e337\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-58fc56757c-jsszw\": pods \"kuadrant-operator-controller-manager-58fc56757c-jsszw\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:28:17.163488 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.163455    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"]
Sep 06 10:28:17.209132 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.209098    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"]
Sep 06 10:28:17.209425 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.209387    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg" podUID="a85b4ebb-9489-43c4-868d-972c5bee7edb" containerName="manager" containerID="cri-o://7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2" gracePeriod=2
Sep 06 10:28:17.216892 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:17.216854    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-jsszw\" is forbidden: User \"system:node:ip-10-0-130-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-109.ec2.internal' and this object" podUID="0bb5af51-9c05-4ed2-8a4a-57e84207e337" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:28:17.226131 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.226100    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"]
Sep 06 10:28:17.258964 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.258867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f47cc-1824-47c8-9c30-f024982c9dd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-sgk48\" (UID: \"795f47cc-1824-47c8-9c30-f024982c9dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:17.258964 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.258904    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbv4\" (UniqueName: \"kubernetes.io/projected/795f47cc-1824-47c8-9c30-f024982c9dd1-kube-api-access-qvbv4\") pod \"kuadrant-operator-controller-manager-58fc56757c-sgk48\" (UID: \"795f47cc-1824-47c8-9c30-f024982c9dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:17.258964 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.258927    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9f7\" (UniqueName: \"kubernetes.io/projected/60610ce6-ad37-4ed1-91b7-747e060cd0f8-kube-api-access-dk9f7\") pod \"kuadrant-operator-controller-manager-68cf589589-rk99v\" (UID: \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:17.259171 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.259033    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60610ce6-ad37-4ed1-91b7-747e060cd0f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-rk99v\" (UID: \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:17.323194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.323171    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:28:17.338957 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:17.338920    2583 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 06 10:28:17.338957 ip-10-0-130-109 kubenswrapper[2583]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-68cf589589-4x2jt_kuadrant-system_d26a16cc-9336-4cc3-84c7-d98156f62d21_0(8a1217e9fcf046840e2ece7255fca6733827bb84cf2a0d50241fa99af3fa2f7a): error adding pod kuadrant-system_kuadrant-operator-controller-manager-68cf589589-4x2jt to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8a1217e9fcf046840e2ece7255fca6733827bb84cf2a0d50241fa99af3fa2f7a" Netns:"/var/run/netns/7772dea6-5c95-4d58-8689-ba2e1ce6c4fd" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-68cf589589-4x2jt;K8S_POD_INFRA_CONTAINER_ID=8a1217e9fcf046840e2ece7255fca6733827bb84cf2a0d50241fa99af3fa2f7a;K8S_POD_UID=d26a16cc-9336-4cc3-84c7-d98156f62d21" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt] networking: [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt/d26a16cc-9336-4cc3-84c7-d98156f62d21:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt 8a1217e9fcf046840e2ece7255fca6733827bb84cf2a0d50241fa99af3fa2f7a network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:17 [10.132.0.23/23]
Sep 06 10:28:17.338957 ip-10-0-130-109 kubenswrapper[2583]:         '
Sep 06 10:28:17.338957 ip-10-0-130-109 kubenswrapper[2583]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 06 10:28:17.338957 ip-10-0-130-109 kubenswrapper[2583]:  >
Sep 06 10:28:17.339164 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:17.338993    2583 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 06 10:28:17.339164 ip-10-0-130-109 kubenswrapper[2583]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-68cf589589-4x2jt_kuadrant-system_d26a16cc-9336-4cc3-84c7-d98156f62d21_0(8a1217e9fcf046840e2ece7255fca6733827bb84cf2a0d50241fa99af3fa2f7a): error adding pod kuadrant-system_kuadrant-operator-controller-manager-68cf589589-4x2jt to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"8a1217e9fcf046840e2ece7255fca6733827bb84cf2a0d50241fa99af3fa2f7a" Netns:"/var/run/netns/7772dea6-5c95-4d58-8689-ba2e1ce6c4fd" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-68cf589589-4x2jt;K8S_POD_INFRA_CONTAINER_ID=8a1217e9fcf046840e2ece7255fca6733827bb84cf2a0d50241fa99af3fa2f7a;K8S_POD_UID=d26a16cc-9336-4cc3-84c7-d98156f62d21" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt] networking: [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt/d26a16cc-9336-4cc3-84c7-d98156f62d21:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt 8a1217e9fcf046840e2ece7255fca6733827bb84cf2a0d50241fa99af3fa2f7a network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:17 [10.132.0.23/23]
Sep 06 10:28:17.339164 ip-10-0-130-109 kubenswrapper[2583]:         '
Sep 06 10:28:17.339164 ip-10-0-130-109 kubenswrapper[2583]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 06 10:28:17.339164 ip-10-0-130-109 kubenswrapper[2583]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:17.360322 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.360288    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9f7\" (UniqueName: \"kubernetes.io/projected/60610ce6-ad37-4ed1-91b7-747e060cd0f8-kube-api-access-dk9f7\") pod \"kuadrant-operator-controller-manager-68cf589589-rk99v\" (UID: \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:17.360504 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.360355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60610ce6-ad37-4ed1-91b7-747e060cd0f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-rk99v\" (UID: \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:17.360504 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.360425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f47cc-1824-47c8-9c30-f024982c9dd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-sgk48\" (UID: \"795f47cc-1824-47c8-9c30-f024982c9dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:17.360504 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.360443    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbv4\" (UniqueName: \"kubernetes.io/projected/795f47cc-1824-47c8-9c30-f024982c9dd1-kube-api-access-qvbv4\") pod \"kuadrant-operator-controller-manager-58fc56757c-sgk48\" (UID: \"795f47cc-1824-47c8-9c30-f024982c9dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:17.360874 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.360850    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60610ce6-ad37-4ed1-91b7-747e060cd0f8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-rk99v\" (UID: \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:17.360941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.360850    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f47cc-1824-47c8-9c30-f024982c9dd1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-sgk48\" (UID: \"795f47cc-1824-47c8-9c30-f024982c9dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:17.368521 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.368485    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbv4\" (UniqueName: \"kubernetes.io/projected/795f47cc-1824-47c8-9c30-f024982c9dd1-kube-api-access-qvbv4\") pod \"kuadrant-operator-controller-manager-58fc56757c-sgk48\" (UID: \"795f47cc-1824-47c8-9c30-f024982c9dd1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:17.368521 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.368514    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9f7\" (UniqueName: \"kubernetes.io/projected/60610ce6-ad37-4ed1-91b7-747e060cd0f8-kube-api-access-dk9f7\") pod \"kuadrant-operator-controller-manager-68cf589589-rk99v\" (UID: \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:17.437757 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.437730    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"
Sep 06 10:28:17.440978 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.440958    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:17.450713 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.450685    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:17.461098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.461073    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bb5af51-9c05-4ed2-8a4a-57e84207e337-extensions-socket-volume\") pod \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\" (UID: \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\") "
Sep 06 10:28:17.461186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.461164    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66f7l\" (UniqueName: \"kubernetes.io/projected/0bb5af51-9c05-4ed2-8a4a-57e84207e337-kube-api-access-66f7l\") pod \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\" (UID: \"0bb5af51-9c05-4ed2-8a4a-57e84207e337\") "
Sep 06 10:28:17.461701 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.461670    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb5af51-9c05-4ed2-8a4a-57e84207e337-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0bb5af51-9c05-4ed2-8a4a-57e84207e337" (UID: "0bb5af51-9c05-4ed2-8a4a-57e84207e337"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:28:17.463397 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.463367    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb5af51-9c05-4ed2-8a4a-57e84207e337-kube-api-access-66f7l" (OuterVolumeSpecName: "kube-api-access-66f7l") pod "0bb5af51-9c05-4ed2-8a4a-57e84207e337" (UID: "0bb5af51-9c05-4ed2-8a4a-57e84207e337"). InnerVolumeSpecName "kube-api-access-66f7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:28:17.561954 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.561911    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wwwv\" (UniqueName: \"kubernetes.io/projected/a85b4ebb-9489-43c4-868d-972c5bee7edb-kube-api-access-7wwwv\") pod \"a85b4ebb-9489-43c4-868d-972c5bee7edb\" (UID: \"a85b4ebb-9489-43c4-868d-972c5bee7edb\") "
Sep 06 10:28:17.562167 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.562150    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bb5af51-9c05-4ed2-8a4a-57e84207e337-extensions-socket-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:28:17.562207 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.562173    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66f7l\" (UniqueName: \"kubernetes.io/projected/0bb5af51-9c05-4ed2-8a4a-57e84207e337-kube-api-access-66f7l\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:28:17.563969 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.563937    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85b4ebb-9489-43c4-868d-972c5bee7edb-kube-api-access-7wwwv" (OuterVolumeSpecName: "kube-api-access-7wwwv") pod "a85b4ebb-9489-43c4-868d-972c5bee7edb" (UID: "a85b4ebb-9489-43c4-868d-972c5bee7edb"). InnerVolumeSpecName "kube-api-access-7wwwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:28:17.662766 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.662722    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wwwv\" (UniqueName: \"kubernetes.io/projected/a85b4ebb-9489-43c4-868d-972c5bee7edb-kube-api-access-7wwwv\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:28:17.716427 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.716277    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"]
Sep 06 10:28:17.721093 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:28:17.721063    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795f47cc_1824_47c8_9c30_f024982c9dd1.slice/crio-b57bd5969ac62f25922bb1db211c1e33edf74fd9429c389057334ea62b93b4d1 WatchSource:0}: Error finding container b57bd5969ac62f25922bb1db211c1e33edf74fd9429c389057334ea62b93b4d1: Status 404 returned error can't find the container with id b57bd5969ac62f25922bb1db211c1e33edf74fd9429c389057334ea62b93b4d1
Sep 06 10:28:17.741330 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.741301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"]
Sep 06 10:28:17.743763 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:28:17.743736    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60610ce6_ad37_4ed1_91b7_747e060cd0f8.slice/crio-3d23b858e134d23584758efdcdbd6acd7fc65884fa54413021243eef7b88707c WatchSource:0}: Error finding container 3d23b858e134d23584758efdcdbd6acd7fc65884fa54413021243eef7b88707c: Status 404 returned error can't find the container with id 3d23b858e134d23584758efdcdbd6acd7fc65884fa54413021243eef7b88707c
Sep 06 10:28:17.889110 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.889062    2583 generic.go:356] "Generic (PLEG): container finished" podID="a85b4ebb-9489-43c4-868d-972c5bee7edb" containerID="7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2" exitCode=0
Sep 06 10:28:17.889285 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.889132    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-j8cpg"
Sep 06 10:28:17.889285 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.889133    2583 scope.go:117] "RemoveContainer" containerID="7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2"
Sep 06 10:28:17.891112 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.891079    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v" event={"ID":"60610ce6-ad37-4ed1-91b7-747e060cd0f8","Type":"ContainerStarted","Data":"e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec"}
Sep 06 10:28:17.891273 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.891188    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:17.891273 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.891218    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v" event={"ID":"60610ce6-ad37-4ed1-91b7-747e060cd0f8","Type":"ContainerStarted","Data":"3d23b858e134d23584758efdcdbd6acd7fc65884fa54413021243eef7b88707c"}
Sep 06 10:28:17.892598 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.892574    2583 generic.go:356] "Generic (PLEG): container finished" podID="0bb5af51-9c05-4ed2-8a4a-57e84207e337" containerID="48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681" exitCode=0
Sep 06 10:28:17.892715 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.892639    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-jsszw"
Sep 06 10:28:17.894689 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.894524    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48" event={"ID":"795f47cc-1824-47c8-9c30-f024982c9dd1","Type":"ContainerStarted","Data":"8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881"}
Sep 06 10:28:17.894689 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.894548    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:17.894689 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.894576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48" event={"ID":"795f47cc-1824-47c8-9c30-f024982c9dd1","Type":"ContainerStarted","Data":"b57bd5969ac62f25922bb1db211c1e33edf74fd9429c389057334ea62b93b4d1"}
Sep 06 10:28:17.894689 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.894647    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:17.898444 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.898424    2583 scope.go:117] "RemoveContainer" containerID="7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2"
Sep 06 10:28:17.898781 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:17.898756    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2\": container with ID starting with 7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2 not found: ID does not exist" containerID="7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2"
Sep 06 10:28:17.898905 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.898788    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2"} err="failed to get container status \"7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2\": rpc error: code = NotFound desc = could not find container \"7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2\": container with ID starting with 7dc0f8f9cc04fd0ae7907552116916569206b93e23013bc0e7262d23c4dbbfe2 not found: ID does not exist"
Sep 06 10:28:17.898905 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.898807    2583 scope.go:117] "RemoveContainer" containerID="48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681"
Sep 06 10:28:17.903910 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.903891    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:17.907565 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.907550    2583 scope.go:117] "RemoveContainer" containerID="48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681"
Sep 06 10:28:17.907865 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:17.907845    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681\": container with ID starting with 48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681 not found: ID does not exist" containerID="48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681"
Sep 06 10:28:17.907913 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.907875    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681"} err="failed to get container status \"48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681\": rpc error: code = NotFound desc = could not find container \"48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681\": container with ID starting with 48cc61ffdd53f2ee35fd558c2f6fe771cc8d6120a9a8d04ac00542a893fa8681 not found: ID does not exist"
Sep 06 10:28:17.915454 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.915407    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v" podStartSLOduration=0.915391385 podStartE2EDuration="915.391385ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:28:17.914013935 +0000 UTC m=+389.186655440" watchObservedRunningTime="2026-09-06 10:28:17.915391385 +0000 UTC m=+389.188032889"
Sep 06 10:28:17.915812 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:17.915783    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" is forbidden: User \"system:node:ip-10-0-130-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-109.ec2.internal' and this object" podUID="d26a16cc-9336-4cc3-84c7-d98156f62d21" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:17.948327 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:17.948270    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48" podStartSLOduration=0.94824605 podStartE2EDuration="948.24605ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:28:17.94685973 +0000 UTC m=+389.219501233" watchObservedRunningTime="2026-09-06 10:28:17.94824605 +0000 UTC m=+389.220887554"
Sep 06 10:28:18.066185 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:18.066082    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpdnz\" (UniqueName: \"kubernetes.io/projected/d26a16cc-9336-4cc3-84c7-d98156f62d21-kube-api-access-dpdnz\") pod \"d26a16cc-9336-4cc3-84c7-d98156f62d21\" (UID: \"d26a16cc-9336-4cc3-84c7-d98156f62d21\") "
Sep 06 10:28:18.066185 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:18.066136    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d26a16cc-9336-4cc3-84c7-d98156f62d21-extensions-socket-volume\") pod \"d26a16cc-9336-4cc3-84c7-d98156f62d21\" (UID: \"d26a16cc-9336-4cc3-84c7-d98156f62d21\") "
Sep 06 10:28:18.066475 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:18.066446    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26a16cc-9336-4cc3-84c7-d98156f62d21-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d26a16cc-9336-4cc3-84c7-d98156f62d21" (UID: "d26a16cc-9336-4cc3-84c7-d98156f62d21"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:28:18.068272 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:18.068245    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26a16cc-9336-4cc3-84c7-d98156f62d21-kube-api-access-dpdnz" (OuterVolumeSpecName: "kube-api-access-dpdnz") pod "d26a16cc-9336-4cc3-84c7-d98156f62d21" (UID: "d26a16cc-9336-4cc3-84c7-d98156f62d21"). InnerVolumeSpecName "kube-api-access-dpdnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:28:18.167724 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:18.167685    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpdnz\" (UniqueName: \"kubernetes.io/projected/d26a16cc-9336-4cc3-84c7-d98156f62d21-kube-api-access-dpdnz\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:28:18.167724 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:18.167718    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d26a16cc-9336-4cc3-84c7-d98156f62d21-extensions-socket-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:28:18.900810 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:18.900776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:18.902923 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:18.902890    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" is forbidden: User \"system:node:ip-10-0-130-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-109.ec2.internal' and this object" podUID="d26a16cc-9336-4cc3-84c7-d98156f62d21" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:18.909001 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:18.908971    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" is forbidden: User \"system:node:ip-10-0-130-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-109.ec2.internal' and this object" podUID="d26a16cc-9336-4cc3-84c7-d98156f62d21" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:19.238992 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:19.238907    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-4x2jt\" is forbidden: User \"system:node:ip-10-0-130-109.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-109.ec2.internal' and this object" podUID="d26a16cc-9336-4cc3-84c7-d98156f62d21" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-4x2jt"
Sep 06 10:28:19.240944 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:19.240919    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb5af51-9c05-4ed2-8a4a-57e84207e337" path="/var/lib/kubelet/pods/0bb5af51-9c05-4ed2-8a4a-57e84207e337/volumes"
Sep 06 10:28:19.241247 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:19.241234    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a85b4ebb-9489-43c4-868d-972c5bee7edb" path="/var/lib/kubelet/pods/a85b4ebb-9489-43c4-868d-972c5bee7edb/volumes"
Sep 06 10:28:19.241586 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:19.241573    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d26a16cc-9336-4cc3-84c7-d98156f62d21" path="/var/lib/kubelet/pods/d26a16cc-9336-4cc3-84c7-d98156f62d21/volumes"
Sep 06 10:28:28.902860 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:28.902827    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:28.903245 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:28.902903    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:28.968938 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:28.968896    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"]
Sep 06 10:28:28.969141 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:28.969112    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48" podUID="795f47cc-1824-47c8-9c30-f024982c9dd1" containerName="manager" containerID="cri-o://8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881" gracePeriod=10
Sep 06 10:28:29.206706 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.206680    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:29.251108 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.251078    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"]
Sep 06 10:28:29.251435 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.251418    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795f47cc-1824-47c8-9c30-f024982c9dd1" containerName="manager"
Sep 06 10:28:29.251507 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.251436    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="795f47cc-1824-47c8-9c30-f024982c9dd1" containerName="manager"
Sep 06 10:28:29.251507 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.251454    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a85b4ebb-9489-43c4-868d-972c5bee7edb" containerName="manager"
Sep 06 10:28:29.251507 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.251463    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a85b4ebb-9489-43c4-868d-972c5bee7edb" containerName="manager"
Sep 06 10:28:29.251660 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.251585    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a85b4ebb-9489-43c4-868d-972c5bee7edb" containerName="manager"
Sep 06 10:28:29.251660 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.251598    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="795f47cc-1824-47c8-9c30-f024982c9dd1" containerName="manager"
Sep 06 10:28:29.254399 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.254379    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:29.268191 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.268150    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"]
Sep 06 10:28:29.355099 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.355060    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvbv4\" (UniqueName: \"kubernetes.io/projected/795f47cc-1824-47c8-9c30-f024982c9dd1-kube-api-access-qvbv4\") pod \"795f47cc-1824-47c8-9c30-f024982c9dd1\" (UID: \"795f47cc-1824-47c8-9c30-f024982c9dd1\") "
Sep 06 10:28:29.355300 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.355156    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f47cc-1824-47c8-9c30-f024982c9dd1-extensions-socket-volume\") pod \"795f47cc-1824-47c8-9c30-f024982c9dd1\" (UID: \"795f47cc-1824-47c8-9c30-f024982c9dd1\") "
Sep 06 10:28:29.355300 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.355241    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkgfh\" (UniqueName: \"kubernetes.io/projected/1dcc0480-e53e-405f-8d89-3168df03059c-kube-api-access-xkgfh\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-r4clp\" (UID: \"1dcc0480-e53e-405f-8d89-3168df03059c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:29.355442 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.355320    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dcc0480-e53e-405f-8d89-3168df03059c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-r4clp\" (UID: \"1dcc0480-e53e-405f-8d89-3168df03059c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:29.355611 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.355587    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795f47cc-1824-47c8-9c30-f024982c9dd1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "795f47cc-1824-47c8-9c30-f024982c9dd1" (UID: "795f47cc-1824-47c8-9c30-f024982c9dd1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:28:29.357115 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.357094    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795f47cc-1824-47c8-9c30-f024982c9dd1-kube-api-access-qvbv4" (OuterVolumeSpecName: "kube-api-access-qvbv4") pod "795f47cc-1824-47c8-9c30-f024982c9dd1" (UID: "795f47cc-1824-47c8-9c30-f024982c9dd1"). InnerVolumeSpecName "kube-api-access-qvbv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:28:29.456070 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.455957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dcc0480-e53e-405f-8d89-3168df03059c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-r4clp\" (UID: \"1dcc0480-e53e-405f-8d89-3168df03059c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:29.456070 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.456036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkgfh\" (UniqueName: \"kubernetes.io/projected/1dcc0480-e53e-405f-8d89-3168df03059c-kube-api-access-xkgfh\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-r4clp\" (UID: \"1dcc0480-e53e-405f-8d89-3168df03059c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:29.456324 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.456088    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/795f47cc-1824-47c8-9c30-f024982c9dd1-extensions-socket-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:28:29.456324 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.456098    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbv4\" (UniqueName: \"kubernetes.io/projected/795f47cc-1824-47c8-9c30-f024982c9dd1-kube-api-access-qvbv4\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:28:29.456493 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.456464    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dcc0480-e53e-405f-8d89-3168df03059c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-r4clp\" (UID: \"1dcc0480-e53e-405f-8d89-3168df03059c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:29.472759 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.472724    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkgfh\" (UniqueName: \"kubernetes.io/projected/1dcc0480-e53e-405f-8d89-3168df03059c-kube-api-access-xkgfh\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-r4clp\" (UID: \"1dcc0480-e53e-405f-8d89-3168df03059c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:29.564833 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.564801    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:29.838883 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.838827    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"]
Sep 06 10:28:29.843064 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:28:29.843036    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcc0480_e53e_405f_8d89_3168df03059c.slice/crio-9400e6cd07fda67a1ad9a60eac9d73d64162a6a677b6455ea68d289dc2c69576 WatchSource:0}: Error finding container 9400e6cd07fda67a1ad9a60eac9d73d64162a6a677b6455ea68d289dc2c69576: Status 404 returned error can't find the container with id 9400e6cd07fda67a1ad9a60eac9d73d64162a6a677b6455ea68d289dc2c69576
Sep 06 10:28:29.953203 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.953168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp" event={"ID":"1dcc0480-e53e-405f-8d89-3168df03059c","Type":"ContainerStarted","Data":"3fbebc155cec47049c001f0fc08d5ac305391303f60a029052283b0b3d754a7b"}
Sep 06 10:28:29.953692 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.953220    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp" event={"ID":"1dcc0480-e53e-405f-8d89-3168df03059c","Type":"ContainerStarted","Data":"9400e6cd07fda67a1ad9a60eac9d73d64162a6a677b6455ea68d289dc2c69576"}
Sep 06 10:28:29.953692 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.953250    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:29.954646 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.954618    2583 generic.go:356] "Generic (PLEG): container finished" podID="795f47cc-1824-47c8-9c30-f024982c9dd1" containerID="8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881" exitCode=0
Sep 06 10:28:29.954766 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.954653    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"
Sep 06 10:28:29.954766 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.954657    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48" event={"ID":"795f47cc-1824-47c8-9c30-f024982c9dd1","Type":"ContainerDied","Data":"8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881"}
Sep 06 10:28:29.954766 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.954702    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48" event={"ID":"795f47cc-1824-47c8-9c30-f024982c9dd1","Type":"ContainerDied","Data":"b57bd5969ac62f25922bb1db211c1e33edf74fd9429c389057334ea62b93b4d1"}
Sep 06 10:28:29.954766 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.954732    2583 scope.go:117] "RemoveContainer" containerID="8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881"
Sep 06 10:28:29.963525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.963507    2583 scope.go:117] "RemoveContainer" containerID="8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881"
Sep 06 10:28:29.963823 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:29.963802    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881\": container with ID starting with 8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881 not found: ID does not exist" containerID="8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881"
Sep 06 10:28:29.963862 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.963833    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881"} err="failed to get container status \"8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881\": rpc error: code = NotFound desc = could not find container \"8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881\": container with ID starting with 8605be855dae8537f6b5587c8b36d8052540793526b877b5f0d78c106ccde881 not found: ID does not exist"
Sep 06 10:28:29.974167 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.974125    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp" podStartSLOduration=0.974108941 podStartE2EDuration="974.108941ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:28:29.972865876 +0000 UTC m=+401.245507379" watchObservedRunningTime="2026-09-06 10:28:29.974108941 +0000 UTC m=+401.246750443"
Sep 06 10:28:29.989393 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.989363    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"]
Sep 06 10:28:29.995062 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:29.995032    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-sgk48"]
Sep 06 10:28:31.239724 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:31.239691    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795f47cc-1824-47c8-9c30-f024982c9dd1" path="/var/lib/kubelet/pods/795f47cc-1824-47c8-9c30-f024982c9dd1/volumes"
Sep 06 10:28:40.967222 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:40.967189    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:28:41.017905 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:41.017873    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"]
Sep 06 10:28:41.018157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:41.018130    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v" podUID="60610ce6-ad37-4ed1-91b7-747e060cd0f8" containerName="manager" containerID="cri-o://e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec" gracePeriod=10
Sep 06 10:28:41.267848 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:41.267822    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:41.340636 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:41.340603    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60610ce6-ad37-4ed1-91b7-747e060cd0f8-extensions-socket-volume\") pod \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\" (UID: \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\") "
Sep 06 10:28:41.340809 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:41.340654    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk9f7\" (UniqueName: \"kubernetes.io/projected/60610ce6-ad37-4ed1-91b7-747e060cd0f8-kube-api-access-dk9f7\") pod \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\" (UID: \"60610ce6-ad37-4ed1-91b7-747e060cd0f8\") "
Sep 06 10:28:41.341033 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:41.341005    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60610ce6-ad37-4ed1-91b7-747e060cd0f8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "60610ce6-ad37-4ed1-91b7-747e060cd0f8" (UID: "60610ce6-ad37-4ed1-91b7-747e060cd0f8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:28:41.342731 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:41.342708    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60610ce6-ad37-4ed1-91b7-747e060cd0f8-kube-api-access-dk9f7" (OuterVolumeSpecName: "kube-api-access-dk9f7") pod "60610ce6-ad37-4ed1-91b7-747e060cd0f8" (UID: "60610ce6-ad37-4ed1-91b7-747e060cd0f8"). InnerVolumeSpecName "kube-api-access-dk9f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:28:41.441370 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:41.441313    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60610ce6-ad37-4ed1-91b7-747e060cd0f8-extensions-socket-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:28:41.441370 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:41.441365    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk9f7\" (UniqueName: \"kubernetes.io/projected/60610ce6-ad37-4ed1-91b7-747e060cd0f8-kube-api-access-dk9f7\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:28:42.012844 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:42.012807    2583 generic.go:356] "Generic (PLEG): container finished" podID="60610ce6-ad37-4ed1-91b7-747e060cd0f8" containerID="e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec" exitCode=0
Sep 06 10:28:42.013254 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:42.012853    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v" event={"ID":"60610ce6-ad37-4ed1-91b7-747e060cd0f8","Type":"ContainerDied","Data":"e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec"}
Sep 06 10:28:42.013254 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:42.012857    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"
Sep 06 10:28:42.013254 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:42.012898    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v" event={"ID":"60610ce6-ad37-4ed1-91b7-747e060cd0f8","Type":"ContainerDied","Data":"3d23b858e134d23584758efdcdbd6acd7fc65884fa54413021243eef7b88707c"}
Sep 06 10:28:42.013254 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:42.012919    2583 scope.go:117] "RemoveContainer" containerID="e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec"
Sep 06 10:28:42.021539 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:42.021321    2583 scope.go:117] "RemoveContainer" containerID="e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec"
Sep 06 10:28:42.021694 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:28:42.021672    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec\": container with ID starting with e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec not found: ID does not exist" containerID="e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec"
Sep 06 10:28:42.021776 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:42.021705    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec"} err="failed to get container status \"e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec\": rpc error: code = NotFound desc = could not find container \"e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec\": container with ID starting with e8044c023974ccc3eb04cee3150b87cbd493622949a2657ecbb099edfab347ec not found: ID does not exist"
Sep 06 10:28:42.040867 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:42.040836    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"]
Sep 06 10:28:42.046570 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:42.046545    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-rk99v"]
Sep 06 10:28:43.241378 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:28:43.241327    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60610ce6-ad37-4ed1-91b7-747e060cd0f8" path="/var/lib/kubelet/pods/60610ce6-ad37-4ed1-91b7-747e060cd0f8/volumes"
Sep 06 10:29:02.401946 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.401900    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"]
Sep 06 10:29:02.402469 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.402174    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60610ce6-ad37-4ed1-91b7-747e060cd0f8" containerName="manager"
Sep 06 10:29:02.402469 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.402185    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="60610ce6-ad37-4ed1-91b7-747e060cd0f8" containerName="manager"
Sep 06 10:29:02.402469 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.402244    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="60610ce6-ad37-4ed1-91b7-747e060cd0f8" containerName="manager"
Sep 06 10:29:02.406808 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.406787    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:02.408944 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.408920    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jx2cd\""
Sep 06 10:29:02.409056 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.408964    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 06 10:29:02.418497 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.418471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"]
Sep 06 10:29:02.511244 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.511206    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/74011f3e-ba14-403f-8b6f-20fba683b951-config-file\") pod \"limitador-limitador-859c4d96d6-f9lxb\" (UID: \"74011f3e-ba14-403f-8b6f-20fba683b951\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:02.511415 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.511351    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkjjg\" (UniqueName: \"kubernetes.io/projected/74011f3e-ba14-403f-8b6f-20fba683b951-kube-api-access-mkjjg\") pod \"limitador-limitador-859c4d96d6-f9lxb\" (UID: \"74011f3e-ba14-403f-8b6f-20fba683b951\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:02.516124 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.516093    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"]
Sep 06 10:29:02.612025 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.611991    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkjjg\" (UniqueName: \"kubernetes.io/projected/74011f3e-ba14-403f-8b6f-20fba683b951-kube-api-access-mkjjg\") pod \"limitador-limitador-859c4d96d6-f9lxb\" (UID: \"74011f3e-ba14-403f-8b6f-20fba683b951\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:02.612199 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.612049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/74011f3e-ba14-403f-8b6f-20fba683b951-config-file\") pod \"limitador-limitador-859c4d96d6-f9lxb\" (UID: \"74011f3e-ba14-403f-8b6f-20fba683b951\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:02.612822 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.612707    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/74011f3e-ba14-403f-8b6f-20fba683b951-config-file\") pod \"limitador-limitador-859c4d96d6-f9lxb\" (UID: \"74011f3e-ba14-403f-8b6f-20fba683b951\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:02.621328 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.621298    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkjjg\" (UniqueName: \"kubernetes.io/projected/74011f3e-ba14-403f-8b6f-20fba683b951-kube-api-access-mkjjg\") pod \"limitador-limitador-859c4d96d6-f9lxb\" (UID: \"74011f3e-ba14-403f-8b6f-20fba683b951\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:02.716444 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.716363    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:02.819648 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.819606    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:29:02.822971 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.822947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:02.832074 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.832046    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:29:02.865975 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.865937    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:29:02.913871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.913837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7e758943-97a7-4ea9-bcc1-6eb56cda5615-config-file\") pod \"limitador-limitador-85d9db4d5d-2n59j\" (UID: \"7e758943-97a7-4ea9-bcc1-6eb56cda5615\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:02.913871 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.913873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4mq4\" (UniqueName: \"kubernetes.io/projected/7e758943-97a7-4ea9-bcc1-6eb56cda5615-kube-api-access-d4mq4\") pod \"limitador-limitador-85d9db4d5d-2n59j\" (UID: \"7e758943-97a7-4ea9-bcc1-6eb56cda5615\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:02.987487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:02.987449    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"]
Sep 06 10:29:02.990034 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:29:02.989998    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74011f3e_ba14_403f_8b6f_20fba683b951.slice/crio-138f3160e0ee938cb22f4c3da3e42ea36fcbf8c9f4ecafc070e7263bf98ef593 WatchSource:0}: Error finding container 138f3160e0ee938cb22f4c3da3e42ea36fcbf8c9f4ecafc070e7263bf98ef593: Status 404 returned error can't find the container with id 138f3160e0ee938cb22f4c3da3e42ea36fcbf8c9f4ecafc070e7263bf98ef593
Sep 06 10:29:03.014273 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:03.014234    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7e758943-97a7-4ea9-bcc1-6eb56cda5615-config-file\") pod \"limitador-limitador-85d9db4d5d-2n59j\" (UID: \"7e758943-97a7-4ea9-bcc1-6eb56cda5615\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:03.014273 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:03.014277    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4mq4\" (UniqueName: \"kubernetes.io/projected/7e758943-97a7-4ea9-bcc1-6eb56cda5615-kube-api-access-d4mq4\") pod \"limitador-limitador-85d9db4d5d-2n59j\" (UID: \"7e758943-97a7-4ea9-bcc1-6eb56cda5615\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:03.014880 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:03.014860    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7e758943-97a7-4ea9-bcc1-6eb56cda5615-config-file\") pod \"limitador-limitador-85d9db4d5d-2n59j\" (UID: \"7e758943-97a7-4ea9-bcc1-6eb56cda5615\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:03.022212 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:03.022186    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4mq4\" (UniqueName: \"kubernetes.io/projected/7e758943-97a7-4ea9-bcc1-6eb56cda5615-kube-api-access-d4mq4\") pod \"limitador-limitador-85d9db4d5d-2n59j\" (UID: \"7e758943-97a7-4ea9-bcc1-6eb56cda5615\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:03.100289 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:03.100250    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb" event={"ID":"74011f3e-ba14-403f-8b6f-20fba683b951","Type":"ContainerStarted","Data":"138f3160e0ee938cb22f4c3da3e42ea36fcbf8c9f4ecafc070e7263bf98ef593"}
Sep 06 10:29:03.133875 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:03.133831    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:03.404447 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:03.404405    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:29:03.408295 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:29:03.408262    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e758943_97a7_4ea9_bcc1_6eb56cda5615.slice/crio-6a15558743e4134fd250c51703b46c0553b363395d9634330ec5c34a2bbbf108 WatchSource:0}: Error finding container 6a15558743e4134fd250c51703b46c0553b363395d9634330ec5c34a2bbbf108: Status 404 returned error can't find the container with id 6a15558743e4134fd250c51703b46c0553b363395d9634330ec5c34a2bbbf108
Sep 06 10:29:04.107599 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:04.107553    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j" event={"ID":"7e758943-97a7-4ea9-bcc1-6eb56cda5615","Type":"ContainerStarted","Data":"6a15558743e4134fd250c51703b46c0553b363395d9634330ec5c34a2bbbf108"}
Sep 06 10:29:05.113662 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:05.113628    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j" event={"ID":"7e758943-97a7-4ea9-bcc1-6eb56cda5615","Type":"ContainerStarted","Data":"ede42dac07d734fcabe3ce96496a7543c42ea1021ae2afee7914f6ce66d214a9"}
Sep 06 10:29:05.114137 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:05.113729    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:05.115210 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:05.115187    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb" event={"ID":"74011f3e-ba14-403f-8b6f-20fba683b951","Type":"ContainerStarted","Data":"6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333"}
Sep 06 10:29:05.115367 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:05.115354    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:05.130147 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:05.130090    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j" podStartSLOduration=2.032040238 podStartE2EDuration="3.130072424s" totalImagesPullingTime="1.098032186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:29:05.127725849 +0000 UTC m=+436.400367351" watchObservedRunningTime="2026-09-06 10:29:05.130072424 +0000 UTC m=+436.402713921"
Sep 06 10:29:05.142482 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:05.142425    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb" podStartSLOduration=1.6235798830000001 podStartE2EDuration="3.142411783s" totalImagesPullingTime="1.5188319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:29:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:29:05.141189138 +0000 UTC m=+436.413830642" watchObservedRunningTime="2026-09-06 10:29:05.142411783 +0000 UTC m=+436.415053286"
Sep 06 10:29:16.120213 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.120129    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:16.120213 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.120198    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"
Sep 06 10:29:16.178555 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.178517    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"]
Sep 06 10:29:16.178729 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.178686    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb" podUID="74011f3e-ba14-403f-8b6f-20fba683b951" containerName="limitador" containerID="cri-o://6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333" gracePeriod=30
Sep 06 10:29:16.718525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.718496    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:16.826178 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.826031    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkjjg\" (UniqueName: \"kubernetes.io/projected/74011f3e-ba14-403f-8b6f-20fba683b951-kube-api-access-mkjjg\") pod \"74011f3e-ba14-403f-8b6f-20fba683b951\" (UID: \"74011f3e-ba14-403f-8b6f-20fba683b951\") "
Sep 06 10:29:16.826375 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.826201    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/74011f3e-ba14-403f-8b6f-20fba683b951-config-file\") pod \"74011f3e-ba14-403f-8b6f-20fba683b951\" (UID: \"74011f3e-ba14-403f-8b6f-20fba683b951\") "
Sep 06 10:29:16.826608 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.826584    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74011f3e-ba14-403f-8b6f-20fba683b951-config-file" (OuterVolumeSpecName: "config-file") pod "74011f3e-ba14-403f-8b6f-20fba683b951" (UID: "74011f3e-ba14-403f-8b6f-20fba683b951"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:29:16.828014 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.827989    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74011f3e-ba14-403f-8b6f-20fba683b951-kube-api-access-mkjjg" (OuterVolumeSpecName: "kube-api-access-mkjjg") pod "74011f3e-ba14-403f-8b6f-20fba683b951" (UID: "74011f3e-ba14-403f-8b6f-20fba683b951"). InnerVolumeSpecName "kube-api-access-mkjjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:29:16.926995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.926958    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkjjg\" (UniqueName: \"kubernetes.io/projected/74011f3e-ba14-403f-8b6f-20fba683b951-kube-api-access-mkjjg\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:29:16.926995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:16.926993    2583 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/74011f3e-ba14-403f-8b6f-20fba683b951-config-file\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:29:17.170786 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.170750    2583 generic.go:356] "Generic (PLEG): container finished" podID="74011f3e-ba14-403f-8b6f-20fba683b951" containerID="6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333" exitCode=0
Sep 06 10:29:17.171186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.170794    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"
Sep 06 10:29:17.171186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.170800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb" event={"ID":"74011f3e-ba14-403f-8b6f-20fba683b951","Type":"ContainerDied","Data":"6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333"}
Sep 06 10:29:17.171186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.170871    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-f9lxb" event={"ID":"74011f3e-ba14-403f-8b6f-20fba683b951","Type":"ContainerDied","Data":"138f3160e0ee938cb22f4c3da3e42ea36fcbf8c9f4ecafc070e7263bf98ef593"}
Sep 06 10:29:17.171186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.170892    2583 scope.go:117] "RemoveContainer" containerID="6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333"
Sep 06 10:29:17.179045 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.179022    2583 scope.go:117] "RemoveContainer" containerID="6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333"
Sep 06 10:29:17.179329 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:29:17.179309    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333\": container with ID starting with 6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333 not found: ID does not exist" containerID="6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333"
Sep 06 10:29:17.179437 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.179352    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333"} err="failed to get container status \"6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333\": rpc error: code = NotFound desc = could not find container \"6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333\": container with ID starting with 6ec0c8a44964a0a2163d38a2f7b332561b75e21a3dc13a5d42b69b2cd1911333 not found: ID does not exist"
Sep 06 10:29:17.190230 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.190197    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"]
Sep 06 10:29:17.196318 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.196291    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-f9lxb"]
Sep 06 10:29:17.240042 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:17.240014    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74011f3e-ba14-403f-8b6f-20fba683b951" path="/var/lib/kubelet/pods/74011f3e-ba14-403f-8b6f-20fba683b951/volumes"
Sep 06 10:29:39.146727 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.146691    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5d4c4495b7-9rhlm"]
Sep 06 10:29:39.147239 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.146978    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74011f3e-ba14-403f-8b6f-20fba683b951" containerName="limitador"
Sep 06 10:29:39.147239 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.146992    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="74011f3e-ba14-403f-8b6f-20fba683b951" containerName="limitador"
Sep 06 10:29:39.147239 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.147059    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="74011f3e-ba14-403f-8b6f-20fba683b951" containerName="limitador"
Sep 06 10:29:39.152194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.152175    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.154376 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.154320    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m7lb4\""
Sep 06 10:29:39.154376 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.154355    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 06 10:29:39.154625 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.154345    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 06 10:29:39.154679 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.154632    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 06 10:29:39.158863 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.158797    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d4c4495b7-9rhlm"]
Sep 06 10:29:39.310950 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.310920    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbhgq\" (UniqueName: \"kubernetes.io/projected/bbce8574-6e42-4c38-9529-ef810d646672-kube-api-access-gbhgq\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.310950 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.310960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-metrics-certs\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.311154 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.311047    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-webhook-cert\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.328857 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.328829    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-5r55h"]
Sep 06 10:29:39.332492 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.332476    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.343531 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.343497    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-5r55h"]
Sep 06 10:29:39.411484 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.411382    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbhgq\" (UniqueName: \"kubernetes.io/projected/bbce8574-6e42-4c38-9529-ef810d646672-kube-api-access-gbhgq\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.411664 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.411533    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-metrics-certs\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.411664 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.411611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-webhook-cert\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.411777 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:29:39.411749    2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 06 10:29:39.412131 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:29:39.412094    2583 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 06 10:29:39.414380 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:29:39.412791    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-metrics-certs podName:bbce8574-6e42-4c38-9529-ef810d646672 nodeName:}" failed. No retries permitted until 2026-09-06 10:29:39.912754367 +0000 UTC m=+471.185395861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-metrics-certs") pod "maas-controller-5d4c4495b7-9rhlm" (UID: "bbce8574-6e42-4c38-9529-ef810d646672") : secret "maas-controller-metrics-tls" not found
Sep 06 10:29:39.414380 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:29:39.413125    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-webhook-cert podName:bbce8574-6e42-4c38-9529-ef810d646672 nodeName:}" failed. No retries permitted until 2026-09-06 10:29:39.912977679 +0000 UTC m=+471.185619174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-webhook-cert") pod "maas-controller-5d4c4495b7-9rhlm" (UID: "bbce8574-6e42-4c38-9529-ef810d646672") : secret "maas-controller-webhook-cert" not found
Sep 06 10:29:39.420212 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.420179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbhgq\" (UniqueName: \"kubernetes.io/projected/bbce8574-6e42-4c38-9529-ef810d646672-kube-api-access-gbhgq\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.512764 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.512721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-metrics-certs\") pod \"maas-controller-6ff9479bc9-5r55h\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") " pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.512764 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.512759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5c4\" (UniqueName: \"kubernetes.io/projected/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-kube-api-access-5w5c4\") pod \"maas-controller-6ff9479bc9-5r55h\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") " pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.512978 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.512908    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-webhook-cert\") pod \"maas-controller-6ff9479bc9-5r55h\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") " pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.613876 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.613833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-webhook-cert\") pod \"maas-controller-6ff9479bc9-5r55h\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") " pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.614059 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.613902    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-metrics-certs\") pod \"maas-controller-6ff9479bc9-5r55h\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") " pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.614059 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.613924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5c4\" (UniqueName: \"kubernetes.io/projected/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-kube-api-access-5w5c4\") pod \"maas-controller-6ff9479bc9-5r55h\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") " pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.616370 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.616310    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-metrics-certs\") pod \"maas-controller-6ff9479bc9-5r55h\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") " pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.616370 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.616365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-webhook-cert\") pod \"maas-controller-6ff9479bc9-5r55h\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") " pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.621260 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.621233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5c4\" (UniqueName: \"kubernetes.io/projected/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-kube-api-access-5w5c4\") pod \"maas-controller-6ff9479bc9-5r55h\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") " pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.642329 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.642291    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:39.911890 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.911869    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-5r55h"]
Sep 06 10:29:39.914628 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:29:39.914601    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97d71e5_238c_41ba_9b19_a5b2c2cc5273.slice/crio-f28aeb9936f17aa041639c5179a5838c6246f4809adb81de91ff0ce9858ed1bc WatchSource:0}: Error finding container f28aeb9936f17aa041639c5179a5838c6246f4809adb81de91ff0ce9858ed1bc: Status 404 returned error can't find the container with id f28aeb9936f17aa041639c5179a5838c6246f4809adb81de91ff0ce9858ed1bc
Sep 06 10:29:39.916373 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.916327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-webhook-cert\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.916473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.916459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-metrics-certs\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.918789 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.918766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-webhook-cert\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:39.918887 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:39.918775    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-metrics-certs\") pod \"maas-controller-5d4c4495b7-9rhlm\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") " pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:40.063554 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:40.063458    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:40.265743 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:40.265709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-5r55h" event={"ID":"b97d71e5-238c-41ba-9b19-a5b2c2cc5273","Type":"ContainerStarted","Data":"f28aeb9936f17aa041639c5179a5838c6246f4809adb81de91ff0ce9858ed1bc"}
Sep 06 10:29:40.336919 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:40.336890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5d4c4495b7-9rhlm"]
Sep 06 10:29:40.339235 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:29:40.339203    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbce8574_6e42_4c38_9529_ef810d646672.slice/crio-0f43fd028ef8f1ea90df89bacc0222e45cf6606f11316d340b6adbc00cdf2beb WatchSource:0}: Error finding container 0f43fd028ef8f1ea90df89bacc0222e45cf6606f11316d340b6adbc00cdf2beb: Status 404 returned error can't find the container with id 0f43fd028ef8f1ea90df89bacc0222e45cf6606f11316d340b6adbc00cdf2beb
Sep 06 10:29:41.273939 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:41.273900    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm" event={"ID":"bbce8574-6e42-4c38-9529-ef810d646672","Type":"ContainerStarted","Data":"0f43fd028ef8f1ea90df89bacc0222e45cf6606f11316d340b6adbc00cdf2beb"}
Sep 06 10:29:43.284880 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:43.284837    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-5r55h" event={"ID":"b97d71e5-238c-41ba-9b19-a5b2c2cc5273","Type":"ContainerStarted","Data":"394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72"}
Sep 06 10:29:43.285374 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:43.284935    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:43.286498 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:43.286468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm" event={"ID":"bbce8574-6e42-4c38-9529-ef810d646672","Type":"ContainerStarted","Data":"46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1"}
Sep 06 10:29:43.286611 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:43.286535    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:43.306610 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:43.306538    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ff9479bc9-5r55h" podStartSLOduration=1.636636616 podStartE2EDuration="4.306522689s" totalImagesPullingTime="2.669886073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:29:43.304983747 +0000 UTC m=+474.577625250" watchObservedRunningTime="2026-09-06 10:29:43.306522689 +0000 UTC m=+474.579164192"
Sep 06 10:29:43.324633 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:43.324582    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm" podStartSLOduration=2.072929287 podStartE2EDuration="4.324566102s" totalImagesPullingTime="2.251636815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:29:43.322794461 +0000 UTC m=+474.595435967" watchObservedRunningTime="2026-09-06 10:29:43.324566102 +0000 UTC m=+474.597207604"
Sep 06 10:29:54.292796 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.292764    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:29:54.293216 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.292837    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:54.356487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.356453    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d4c4495b7-9rhlm"]
Sep 06 10:29:54.356687 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.356656    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm" podUID="bbce8574-6e42-4c38-9529-ef810d646672" containerName="manager" containerID="cri-o://46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1" gracePeriod=30
Sep 06 10:29:54.598193 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.598169    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:54.623919 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.623888    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbhgq\" (UniqueName: \"kubernetes.io/projected/bbce8574-6e42-4c38-9529-ef810d646672-kube-api-access-gbhgq\") pod \"bbce8574-6e42-4c38-9529-ef810d646672\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") "
Sep 06 10:29:54.624062 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.623939    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-metrics-certs\") pod \"bbce8574-6e42-4c38-9529-ef810d646672\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") "
Sep 06 10:29:54.624062 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.623958    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-webhook-cert\") pod \"bbce8574-6e42-4c38-9529-ef810d646672\" (UID: \"bbce8574-6e42-4c38-9529-ef810d646672\") "
Sep 06 10:29:54.626231 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.626188    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bbce8574-6e42-4c38-9529-ef810d646672" (UID: "bbce8574-6e42-4c38-9529-ef810d646672"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:29:54.626400 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.626199    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbce8574-6e42-4c38-9529-ef810d646672-kube-api-access-gbhgq" (OuterVolumeSpecName: "kube-api-access-gbhgq") pod "bbce8574-6e42-4c38-9529-ef810d646672" (UID: "bbce8574-6e42-4c38-9529-ef810d646672"). InnerVolumeSpecName "kube-api-access-gbhgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:29:54.626475 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.626401    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bbce8574-6e42-4c38-9529-ef810d646672" (UID: "bbce8574-6e42-4c38-9529-ef810d646672"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:29:54.725012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.724977    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:29:54.725012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.725006    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bbce8574-6e42-4c38-9529-ef810d646672-webhook-cert\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:29:54.725012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:54.725015    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbhgq\" (UniqueName: \"kubernetes.io/projected/bbce8574-6e42-4c38-9529-ef810d646672-kube-api-access-gbhgq\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:29:55.343661 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:55.343630    2583 generic.go:356] "Generic (PLEG): container finished" podID="bbce8574-6e42-4c38-9529-ef810d646672" containerID="46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1" exitCode=0
Sep 06 10:29:55.344077 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:55.343668    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm" event={"ID":"bbce8574-6e42-4c38-9529-ef810d646672","Type":"ContainerDied","Data":"46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1"}
Sep 06 10:29:55.344077 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:55.343678    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm"
Sep 06 10:29:55.344077 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:55.343707    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5d4c4495b7-9rhlm" event={"ID":"bbce8574-6e42-4c38-9529-ef810d646672","Type":"ContainerDied","Data":"0f43fd028ef8f1ea90df89bacc0222e45cf6606f11316d340b6adbc00cdf2beb"}
Sep 06 10:29:55.344077 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:55.343729    2583 scope.go:117] "RemoveContainer" containerID="46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1"
Sep 06 10:29:55.351938 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:55.351914    2583 scope.go:117] "RemoveContainer" containerID="46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1"
Sep 06 10:29:55.352248 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:29:55.352224    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1\": container with ID starting with 46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1 not found: ID does not exist" containerID="46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1"
Sep 06 10:29:55.352308 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:55.352258    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1"} err="failed to get container status \"46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1\": rpc error: code = NotFound desc = could not find container \"46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1\": container with ID starting with 46182d02171c5fde02843f0bcecf9a0e71f00709cc49664091e6afdabcf186a1 not found: ID does not exist"
Sep 06 10:29:55.360054 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:55.360024    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5d4c4495b7-9rhlm"]
Sep 06 10:29:55.364544 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:55.364522    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5d4c4495b7-9rhlm"]
Sep 06 10:29:57.239050 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:29:57.239015    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbce8574-6e42-4c38-9529-ef810d646672" path="/var/lib/kubelet/pods/bbce8574-6e42-4c38-9529-ef810d646672/volumes"
Sep 06 10:30:16.871746 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.871708    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-m767q"]
Sep 06 10:30:16.872216 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.871991    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbce8574-6e42-4c38-9529-ef810d646672" containerName="manager"
Sep 06 10:30:16.872216 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.872002    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbce8574-6e42-4c38-9529-ef810d646672" containerName="manager"
Sep 06 10:30:16.872216 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.872086    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbce8574-6e42-4c38-9529-ef810d646672" containerName="manager"
Sep 06 10:30:16.877758 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.877736    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:16.880012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.879988    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 10:30:16.880564 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.880543    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 06 10:30:16.880648 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.880570    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 10:30:16.880648 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.880545    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 10:30:16.880847 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.880834    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-965kn\""
Sep 06 10:30:16.891952 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.891925    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-m767q"]
Sep 06 10:30:16.995791 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.995746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-maas-api-tls\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:16.995977 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.995837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5f7374-5300-400d-adc7-f47a19046797-openshift-service-ca\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:16.995977 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.995876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2q62\" (UniqueName: \"kubernetes.io/projected/4e5f7374-5300-400d-adc7-f47a19046797-kube-api-access-p2q62\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:16.995977 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:16.995912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-metrics-certs\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.097064 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.097030    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-metrics-certs\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.097260 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.097089    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-maas-api-tls\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.097260 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.097126    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5f7374-5300-400d-adc7-f47a19046797-openshift-service-ca\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.097260 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.097154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2q62\" (UniqueName: \"kubernetes.io/projected/4e5f7374-5300-400d-adc7-f47a19046797-kube-api-access-p2q62\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.097875 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.097850    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5f7374-5300-400d-adc7-f47a19046797-openshift-service-ca\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.099637 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.099613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-metrics-certs\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.099756 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.099734    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-maas-api-tls\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.105646 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.105619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2q62\" (UniqueName: \"kubernetes.io/projected/4e5f7374-5300-400d-adc7-f47a19046797-kube-api-access-p2q62\") pod \"maas-api-679c995ddf-m767q\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") " pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.188101 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.188011    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:17.468451 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.468371    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"]
Sep 06 10:30:17.472782 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.472755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:17.473726 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.473669    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-m767q"]
Sep 06 10:30:17.479054 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.475521    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 06 10:30:17.479054 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.475524    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-94hsd\""
Sep 06 10:30:17.486177 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.486074    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"]
Sep 06 10:30:17.500623 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.500596    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkktk\" (UniqueName: \"kubernetes.io/projected/ce415c4a-d418-4975-bc07-d2257bc91f31-kube-api-access-rkktk\") pod \"payload-pre-processing-59fbc47bf8-96c5t\" (UID: \"ce415c4a-d418-4975-bc07-d2257bc91f31\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:17.500754 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.500639    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce415c4a-d418-4975-bc07-d2257bc91f31-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-96c5t\" (UID: \"ce415c4a-d418-4975-bc07-d2257bc91f31\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:17.512539 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.512513    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:30:17.601400 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.601367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkktk\" (UniqueName: \"kubernetes.io/projected/ce415c4a-d418-4975-bc07-d2257bc91f31-kube-api-access-rkktk\") pod \"payload-pre-processing-59fbc47bf8-96c5t\" (UID: \"ce415c4a-d418-4975-bc07-d2257bc91f31\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:17.601600 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.601421    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce415c4a-d418-4975-bc07-d2257bc91f31-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-96c5t\" (UID: \"ce415c4a-d418-4975-bc07-d2257bc91f31\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:17.602465 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.602438    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce415c4a-d418-4975-bc07-d2257bc91f31-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-96c5t\" (UID: \"ce415c4a-d418-4975-bc07-d2257bc91f31\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:17.611069 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.611036    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkktk\" (UniqueName: \"kubernetes.io/projected/ce415c4a-d418-4975-bc07-d2257bc91f31-kube-api-access-rkktk\") pod \"payload-pre-processing-59fbc47bf8-96c5t\" (UID: \"ce415c4a-d418-4975-bc07-d2257bc91f31\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:17.793728 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:17.793627    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:18.068793 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:18.068692    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"]
Sep 06 10:30:18.071149 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:30:18.071118    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce415c4a_d418_4975_bc07_d2257bc91f31.slice/crio-d386d5f13088db814735baabb80639009ab67db31cce1d8479e029d1bfb49933 WatchSource:0}: Error finding container d386d5f13088db814735baabb80639009ab67db31cce1d8479e029d1bfb49933: Status 404 returned error can't find the container with id d386d5f13088db814735baabb80639009ab67db31cce1d8479e029d1bfb49933
Sep 06 10:30:18.446319 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:18.446270    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q" event={"ID":"4e5f7374-5300-400d-adc7-f47a19046797","Type":"ContainerStarted","Data":"51e2954988074c12556f4bca2ece51b08d8c74724509191f09d83adb8134ccff"}
Sep 06 10:30:18.447622 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:18.447586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t" event={"ID":"ce415c4a-d418-4975-bc07-d2257bc91f31","Type":"ContainerStarted","Data":"d386d5f13088db814735baabb80639009ab67db31cce1d8479e029d1bfb49933"}
Sep 06 10:30:19.454491 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:19.454446    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q" event={"ID":"4e5f7374-5300-400d-adc7-f47a19046797","Type":"ContainerStarted","Data":"311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7"}
Sep 06 10:30:19.454917 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:19.454514    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:19.470041 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:19.469993    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q" podStartSLOduration=2.091150776 podStartE2EDuration="3.469978151s" totalImagesPullingTime="1.378827375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:19.468901261 +0000 UTC m=+510.741542764" watchObservedRunningTime="2026-09-06 10:30:19.469978151 +0000 UTC m=+510.742619653"
Sep 06 10:30:20.460353 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:20.460235    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t" event={"ID":"ce415c4a-d418-4975-bc07-d2257bc91f31","Type":"ContainerStarted","Data":"6b262cab94759eb96d1f18ca5e962a2a37b218cf749f86d2a51aeae4a71e88a4"}
Sep 06 10:30:20.475032 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:20.474951    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t" podStartSLOduration=1.422454252 podStartE2EDuration="3.474934563s" totalImagesPullingTime="2.052480311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:20.474486165 +0000 UTC m=+511.747127669" watchObservedRunningTime="2026-09-06 10:30:20.474934563 +0000 UTC m=+511.747576066"
Sep 06 10:30:21.464912 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:21.464881    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:25.466132 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:25.466093    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:30:32.477524 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:32.477491    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-96c5t"
Sep 06 10:30:35.501166 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.501133    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"]
Sep 06 10:30:35.504440 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.504416    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.506913 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.506885    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 06 10:30:35.506913 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.506910    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 06 10:30:35.507114 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.506891    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zm5kx\""
Sep 06 10:30:35.507114 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.506893    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 06 10:30:35.515165 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.515137    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"]
Sep 06 10:30:35.547741 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.547706    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.547920 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.547760    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.547920 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.547818    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqgzp\" (UniqueName: \"kubernetes.io/projected/d67fdca1-851d-42b3-bcba-3ebe5ba27351-kube-api-access-jqgzp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.547920 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.547858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67fdca1-851d-42b3-bcba-3ebe5ba27351-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.547920 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.547885    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.547920 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.547902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.648502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.648466    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.648698 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.648508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.648698 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.648580    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.648881 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.648854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.649006 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.648918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqgzp\" (UniqueName: \"kubernetes.io/projected/d67fdca1-851d-42b3-bcba-3ebe5ba27351-kube-api-access-jqgzp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.649006 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.648954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.649006 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.648959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67fdca1-851d-42b3-bcba-3ebe5ba27351-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.649177 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.649017    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.649177 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.648959    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.651180 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.651153    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d67fdca1-851d-42b3-bcba-3ebe5ba27351-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.651427 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.651411    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d67fdca1-851d-42b3-bcba-3ebe5ba27351-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.660318 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.660285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqgzp\" (UniqueName: \"kubernetes.io/projected/d67fdca1-851d-42b3-bcba-3ebe5ba27351-kube-api-access-jqgzp\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4\" (UID: \"d67fdca1-851d-42b3-bcba-3ebe5ba27351\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:35.816573 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:35.816478    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:36.088846 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:36.088818    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"]
Sep 06 10:30:36.091161 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:30:36.091133    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67fdca1_851d_42b3_bcba_3ebe5ba27351.slice/crio-fe399264f8010471731c8f7d2630fc82e493ec19870737736ee6ba7ed9c4715b WatchSource:0}: Error finding container fe399264f8010471731c8f7d2630fc82e493ec19870737736ee6ba7ed9c4715b: Status 404 returned error can't find the container with id fe399264f8010471731c8f7d2630fc82e493ec19870737736ee6ba7ed9c4715b
Sep 06 10:30:36.170005 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:36.169961    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:30:36.539410 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:36.539298    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4" event={"ID":"d67fdca1-851d-42b3-bcba-3ebe5ba27351","Type":"ContainerStarted","Data":"fe399264f8010471731c8f7d2630fc82e493ec19870737736ee6ba7ed9c4715b"}
Sep 06 10:30:37.545155 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:37.545061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4" event={"ID":"d67fdca1-851d-42b3-bcba-3ebe5ba27351","Type":"ContainerStarted","Data":"f512dd30b99f5491762996b5a0aa4bad605d2e4eddb4f6aaba8680d752e17baa"}
Sep 06 10:30:37.562778 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:37.562731    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4" podStartSLOduration=1.474925877 podStartE2EDuration="2.562715286s" totalImagesPullingTime="1.087789409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:37.561135264 +0000 UTC m=+528.833776795" watchObservedRunningTime="2026-09-06 10:30:37.562715286 +0000 UTC m=+528.835356789"
Sep 06 10:30:42.084942 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.084864    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"]
Sep 06 10:30:42.092785 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.092760    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.095439 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.095416    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 06 10:30:42.101098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.101071    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"]
Sep 06 10:30:42.204943 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.204902    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.205145 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.204953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpt6r\" (UniqueName: \"kubernetes.io/projected/9eefec55-0993-4ba7-ac4b-33731e9553d9-kube-api-access-cpt6r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.205145 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.204989    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.205145 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.205011    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.205145 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.205026    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eefec55-0993-4ba7-ac4b-33731e9553d9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.205145 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.205052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.305684 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.305649    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.305872 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.305697    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpt6r\" (UniqueName: \"kubernetes.io/projected/9eefec55-0993-4ba7-ac4b-33731e9553d9-kube-api-access-cpt6r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.305872 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.305723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.305872 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.305742    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.305872 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.305757    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eefec55-0993-4ba7-ac4b-33731e9553d9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.305872 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.305774    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.306140 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.306114    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.306193 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.306146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.306243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.306204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.308059 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.308033    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eefec55-0993-4ba7-ac4b-33731e9553d9-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.308283 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.308263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eefec55-0993-4ba7-ac4b-33731e9553d9-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.315053 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.315030    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpt6r\" (UniqueName: \"kubernetes.io/projected/9eefec55-0993-4ba7-ac4b-33731e9553d9-kube-api-access-cpt6r\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zbx4l\" (UID: \"9eefec55-0993-4ba7-ac4b-33731e9553d9\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.404824 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.404730    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:42.678986 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.678795    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"]
Sep 06 10:30:42.681835 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:30:42.681804    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eefec55_0993_4ba7_ac4b_33731e9553d9.slice/crio-02945dae45365db3b8e0785e6da079b48fd5f1b7d00692d5f38a914340968095 WatchSource:0}: Error finding container 02945dae45365db3b8e0785e6da079b48fd5f1b7d00692d5f38a914340968095: Status 404 returned error can't find the container with id 02945dae45365db3b8e0785e6da079b48fd5f1b7d00692d5f38a914340968095
Sep 06 10:30:42.868278 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:42.868250    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:30:43.578771 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:43.578738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l" event={"ID":"9eefec55-0993-4ba7-ac4b-33731e9553d9","Type":"ContainerStarted","Data":"071a4e25621201e1f57e04a6a451116898008cb0525a51ac54e610954c21edf4"}
Sep 06 10:30:43.579155 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:43.578787    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l" event={"ID":"9eefec55-0993-4ba7-ac4b-33731e9553d9","Type":"ContainerStarted","Data":"02945dae45365db3b8e0785e6da079b48fd5f1b7d00692d5f38a914340968095"}
Sep 06 10:30:43.598893 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:43.598834    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l" podStartSLOduration=1.434194029 podStartE2EDuration="1.598814856s" totalImagesPullingTime="164.620827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:43.596710474 +0000 UTC m=+534.869351977" watchObservedRunningTime="2026-09-06 10:30:43.598814856 +0000 UTC m=+534.871456359"
Sep 06 10:30:45.817065 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:45.817023    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:45.817487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:45.817129    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:45.829501 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:45.829466    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:46.602952 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:46.602919    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4"
Sep 06 10:30:52.405127 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:52.405085    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:52.405597 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:52.405148    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:52.417846 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:52.417816    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:52.628095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:52.628057    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zbx4l"
Sep 06 10:30:54.871282 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:30:54.871242    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:31:04.430589 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:04.430551    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:31:20.780175 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:20.780134    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:31:21.295470 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.295436    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"]
Sep 06 10:31:21.298938 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.298910    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.301202 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.301171    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 06 10:31:21.309862 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.309831    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"]
Sep 06 10:31:21.349169 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.349119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.349169 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.349175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.349435 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.349272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.349435 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.349312    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.349435 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.349362    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p725f\" (UniqueName: \"kubernetes.io/projected/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-kube-api-access-p725f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.349435 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.349406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.450495 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.450445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.450671 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.450600    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.450671 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.450627    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.450671 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.450664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.450832 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.450808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.450889 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.450854    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p725f\" (UniqueName: \"kubernetes.io/projected/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-kube-api-access-p725f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.451011 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.450988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.451070 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.451010    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.451205 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.451182    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.452855 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.452837    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.452947 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.452930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.457927 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.457898    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p725f\" (UniqueName: \"kubernetes.io/projected/ccf8260b-71ec-48d6-a05a-e0ada8b696f5-kube-api-access-p725f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5vgdz\" (UID: \"ccf8260b-71ec-48d6-a05a-e0ada8b696f5\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.610587 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.610544    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:21.885262 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:21.885230    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"]
Sep 06 10:31:21.887732 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:31:21.887691    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf8260b_71ec_48d6_a05a_e0ada8b696f5.slice/crio-09b897836654bb8be49c262d264f52ca3f507e5eea87086af7252d2be13a8bc7 WatchSource:0}: Error finding container 09b897836654bb8be49c262d264f52ca3f507e5eea87086af7252d2be13a8bc7: Status 404 returned error can't find the container with id 09b897836654bb8be49c262d264f52ca3f507e5eea87086af7252d2be13a8bc7
Sep 06 10:31:22.749569 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:22.749533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz" event={"ID":"ccf8260b-71ec-48d6-a05a-e0ada8b696f5","Type":"ContainerStarted","Data":"83a36c12a8e2e5dec1977ec74bd6af15d19e7336f1abafcf1e9979058e65e5d4"}
Sep 06 10:31:22.749741 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:22.749586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz" event={"ID":"ccf8260b-71ec-48d6-a05a-e0ada8b696f5","Type":"ContainerStarted","Data":"09b897836654bb8be49c262d264f52ca3f507e5eea87086af7252d2be13a8bc7"}
Sep 06 10:31:22.766980 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:22.766922    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz" podStartSLOduration=1.608201233 podStartE2EDuration="1.766903673s" totalImagesPullingTime="158.70244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:22.765150534 +0000 UTC m=+574.037792061" watchObservedRunningTime="2026-09-06 10:31:22.766903673 +0000 UTC m=+574.039545225"
Sep 06 10:31:26.170594 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:26.170554    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:31:31.276580 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:31.276545    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:31:31.610895 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:31.610853    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:31.611072 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:31.610943    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:31.623509 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:31.623475    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:31:31.800878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:31:31.800844    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5vgdz"
Sep 06 10:32:53.894989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:53.894952    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:32:54.806712 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.806675    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"]
Sep 06 10:32:54.810257 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.810232    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.812221 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.812199    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-04526a-serving-cert\""
Sep 06 10:32:54.823505 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.823477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.823657 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.823564    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-metrics-certs\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.823657 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.823621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2nx8\" (UniqueName: \"kubernetes.io/projected/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-kube-api-access-l2nx8\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.823764 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.823660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-maas-api-tls\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.824323 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.824284    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"]
Sep 06 10:32:54.924243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.924201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2nx8\" (UniqueName: \"kubernetes.io/projected/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-kube-api-access-l2nx8\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.924243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.924259    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-maas-api-tls\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.924761 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.924372    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.924761 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.924397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-metrics-certs\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.924761 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:32:54.924448    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-serving-cert: secret "maas-api-e2e-inf-a-04526a-serving-cert" not found
Sep 06 10:32:54.924761 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:32:54.924529    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-maas-api-tls podName:27d6fbc2-2eb7-4403-973b-6b63ef7d188b nodeName:}" failed. No retries permitted until 2026-09-06 10:32:55.424505983 +0000 UTC m=+666.697147481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-maas-api-tls") pod "maas-api-e2e-inf-a-04526a-67d748f585-mb2rm" (UID: "27d6fbc2-2eb7-4403-973b-6b63ef7d188b") : secret "maas-api-e2e-inf-a-04526a-serving-cert" not found
Sep 06 10:32:54.925248 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.925208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.927069 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.927043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-metrics-certs\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:54.933804 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:54.933779    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2nx8\" (UniqueName: \"kubernetes.io/projected/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-kube-api-access-l2nx8\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:55.429063 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.429028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-maas-api-tls\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:55.429858 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.429829    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"]
Sep 06 10:32:55.431484 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.431461    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-maas-api-tls\") pod \"maas-api-e2e-inf-a-04526a-67d748f585-mb2rm\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:55.433081 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.433064    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:32:55.435537 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.435515    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-04526a\""
Sep 06 10:32:55.435658 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.435637    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-04526a-dockercfg-r54cf\""
Sep 06 10:32:55.445275 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.445247    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"]
Sep 06 10:32:55.474000 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.473952    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"]
Sep 06 10:32:55.477033 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.477014    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:32:55.484712 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.484683    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"]
Sep 06 10:32:55.530014 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.529981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp2w9\" (UniqueName: \"kubernetes.io/projected/3725842c-df7c-4b74-b6e5-6103c49f8675-kube-api-access-pp2w9\") pod \"payload-processing-e2e-inf-a-04526a-596594d78f-jp227\" (UID: \"3725842c-df7c-4b74-b6e5-6103c49f8675\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:32:55.530190 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.530021    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3725842c-df7c-4b74-b6e5-6103c49f8675-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04526a-596594d78f-jp227\" (UID: \"3725842c-df7c-4b74-b6e5-6103c49f8675\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:32:55.530190 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.530055    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g\" (UID: \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:32:55.530190 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.530110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqzq\" (UniqueName: \"kubernetes.io/projected/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-kube-api-access-fhqzq\") pod \"payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g\" (UID: \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:32:55.630519 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.630479    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqzq\" (UniqueName: \"kubernetes.io/projected/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-kube-api-access-fhqzq\") pod \"payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g\" (UID: \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:32:55.630691 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.630552    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp2w9\" (UniqueName: \"kubernetes.io/projected/3725842c-df7c-4b74-b6e5-6103c49f8675-kube-api-access-pp2w9\") pod \"payload-processing-e2e-inf-a-04526a-596594d78f-jp227\" (UID: \"3725842c-df7c-4b74-b6e5-6103c49f8675\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:32:55.630691 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.630584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3725842c-df7c-4b74-b6e5-6103c49f8675-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04526a-596594d78f-jp227\" (UID: \"3725842c-df7c-4b74-b6e5-6103c49f8675\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:32:55.630691 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.630612    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g\" (UID: \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:32:55.631370 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.631328    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g\" (UID: \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:32:55.631370 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.631327    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3725842c-df7c-4b74-b6e5-6103c49f8675-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-04526a-596594d78f-jp227\" (UID: \"3725842c-df7c-4b74-b6e5-6103c49f8675\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:32:55.640011 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.639981    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp2w9\" (UniqueName: \"kubernetes.io/projected/3725842c-df7c-4b74-b6e5-6103c49f8675-kube-api-access-pp2w9\") pod \"payload-processing-e2e-inf-a-04526a-596594d78f-jp227\" (UID: \"3725842c-df7c-4b74-b6e5-6103c49f8675\") " pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:32:55.640165 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.640140    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqzq\" (UniqueName: \"kubernetes.io/projected/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-kube-api-access-fhqzq\") pod \"payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g\" (UID: \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:32:55.721366 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.721270    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:55.748432 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.748397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:32:55.789020 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.788735    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:32:55.994672 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.994646    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"]
Sep 06 10:32:55.997282 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:32:55.997256    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d6fbc2_2eb7_4403_973b_6b63ef7d188b.slice/crio-fce7f6dbf9d681e5e5ccfaf39bffa66de31e24622da4e04522139f645a5aa777 WatchSource:0}: Error finding container fce7f6dbf9d681e5e5ccfaf39bffa66de31e24622da4e04522139f645a5aa777: Status 404 returned error can't find the container with id fce7f6dbf9d681e5e5ccfaf39bffa66de31e24622da4e04522139f645a5aa777
Sep 06 10:32:55.998705 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:55.998686    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:32:56.031546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:56.031471    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"]
Sep 06 10:32:56.035425 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:32:56.035395    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c88fab_1746_46eb_b51a_1b8d1dd4e485.slice/crio-3d3a2202d775be6d2dd7e76f3d0593c8c5195010d76e4f5c5bcc5e386f6a8c92 WatchSource:0}: Error finding container 3d3a2202d775be6d2dd7e76f3d0593c8c5195010d76e4f5c5bcc5e386f6a8c92: Status 404 returned error can't find the container with id 3d3a2202d775be6d2dd7e76f3d0593c8c5195010d76e4f5c5bcc5e386f6a8c92
Sep 06 10:32:56.082634 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:56.082608    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"]
Sep 06 10:32:56.085244 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:32:56.085211    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3725842c_df7c_4b74_b6e5_6103c49f8675.slice/crio-ac7f7095e0555fdc19bc19d1d28399f4b9b646c0b634ad1c664b26a63871f8c4 WatchSource:0}: Error finding container ac7f7095e0555fdc19bc19d1d28399f4b9b646c0b634ad1c664b26a63871f8c4: Status 404 returned error can't find the container with id ac7f7095e0555fdc19bc19d1d28399f4b9b646c0b634ad1c664b26a63871f8c4
Sep 06 10:32:56.157610 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:56.157576    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227" event={"ID":"3725842c-df7c-4b74-b6e5-6103c49f8675","Type":"ContainerStarted","Data":"ac7f7095e0555fdc19bc19d1d28399f4b9b646c0b634ad1c664b26a63871f8c4"}
Sep 06 10:32:56.158928 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:56.158899    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm" event={"ID":"27d6fbc2-2eb7-4403-973b-6b63ef7d188b","Type":"ContainerStarted","Data":"fce7f6dbf9d681e5e5ccfaf39bffa66de31e24622da4e04522139f645a5aa777"}
Sep 06 10:32:56.160787 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:56.160751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g" event={"ID":"f4c88fab-1746-46eb-b51a-1b8d1dd4e485","Type":"ContainerStarted","Data":"57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d"}
Sep 06 10:32:56.160903 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:56.160800    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g" event={"ID":"f4c88fab-1746-46eb-b51a-1b8d1dd4e485","Type":"ContainerStarted","Data":"3d3a2202d775be6d2dd7e76f3d0593c8c5195010d76e4f5c5bcc5e386f6a8c92"}
Sep 06 10:32:56.160903 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:56.160888    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:32:56.177998 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:56.177945    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g" podStartSLOduration=1.177927451 podStartE2EDuration="1.177927451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:32:56.175668812 +0000 UTC m=+667.448310321" watchObservedRunningTime="2026-09-06 10:32:56.177927451 +0000 UTC m=+667.450568945"
Sep 06 10:32:57.167466 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:57.167429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm" event={"ID":"27d6fbc2-2eb7-4403-973b-6b63ef7d188b","Type":"ContainerStarted","Data":"bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6"}
Sep 06 10:32:57.167921 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:57.167530    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:32:57.169050 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:57.169021    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227" event={"ID":"3725842c-df7c-4b74-b6e5-6103c49f8675","Type":"ContainerStarted","Data":"de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d"}
Sep 06 10:32:57.186810 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:57.186755    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm" podStartSLOduration=2.773195169 podStartE2EDuration="3.186736225s" totalImagesPullingTime="413.541056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:32:57.184690792 +0000 UTC m=+668.457332306" watchObservedRunningTime="2026-09-06 10:32:57.186736225 +0000 UTC m=+668.459377728"
Sep 06 10:32:57.200259 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:57.200211    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227" podStartSLOduration=2.200191082 podStartE2EDuration="2.200191082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:32:57.19962787 +0000 UTC m=+668.472269374" watchObservedRunningTime="2026-09-06 10:32:57.200191082 +0000 UTC m=+668.472832585"
Sep 06 10:32:58.173893 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:32:58.173858    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:33:02.819093 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.819055    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"]
Sep 06 10:33:02.822329 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.822304    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:02.825574 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.825552    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dbaf75b1-dockercfg-sqpqk\""
Sep 06 10:33:02.825849 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.825827    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dbaf75b1\""
Sep 06 10:33:02.843801 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.843767    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"]
Sep 06 10:33:02.878815 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.878777    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"]
Sep 06 10:33:02.882019 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.882000    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:02.892646 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.892617    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"]
Sep 06 10:33:02.901945 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.901910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb474364-0545-48f7-ad03-76467b42878c-plugins-config-volume\") pod \"payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k\" (UID: \"eb474364-0545-48f7-ad03-76467b42878c\") " pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:02.902097 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:02.901971    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4mm2\" (UniqueName: \"kubernetes.io/projected/eb474364-0545-48f7-ad03-76467b42878c-kube-api-access-k4mm2\") pod \"payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k\" (UID: \"eb474364-0545-48f7-ad03-76467b42878c\") " pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:03.002574 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.002538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4mm2\" (UniqueName: \"kubernetes.io/projected/eb474364-0545-48f7-ad03-76467b42878c-kube-api-access-k4mm2\") pod \"payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k\" (UID: \"eb474364-0545-48f7-ad03-76467b42878c\") " pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:03.002780 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.002636    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb474364-0545-48f7-ad03-76467b42878c-plugins-config-volume\") pod \"payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k\" (UID: \"eb474364-0545-48f7-ad03-76467b42878c\") " pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:03.002780 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.002676    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrkh\" (UniqueName: \"kubernetes.io/projected/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-kube-api-access-fwrkh\") pod \"payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf\" (UID: \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:03.002780 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.002708    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf\" (UID: \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:03.003372 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.003329    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb474364-0545-48f7-ad03-76467b42878c-plugins-config-volume\") pod \"payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k\" (UID: \"eb474364-0545-48f7-ad03-76467b42878c\") " pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:03.010278 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.010247    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4mm2\" (UniqueName: \"kubernetes.io/projected/eb474364-0545-48f7-ad03-76467b42878c-kube-api-access-k4mm2\") pod \"payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k\" (UID: \"eb474364-0545-48f7-ad03-76467b42878c\") " pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:03.103386 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.103327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf\" (UID: \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:03.103553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.103484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrkh\" (UniqueName: \"kubernetes.io/projected/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-kube-api-access-fwrkh\") pod \"payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf\" (UID: \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:03.104012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.103990    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf\" (UID: \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:03.112412 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.112389    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrkh\" (UniqueName: \"kubernetes.io/projected/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-kube-api-access-fwrkh\") pod \"payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf\" (UID: \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:03.134012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.133969    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:03.180008 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.179976    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:33:03.192259 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.191971    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:03.405650 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.405620    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"]
Sep 06 10:33:03.407410 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:33:03.407385    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb474364_0545_48f7_ad03_76467b42878c.slice/crio-34f83950b790ecdac2f8375f075fe290a3da2dbceedc8a7d74a94793599cbba1 WatchSource:0}: Error finding container 34f83950b790ecdac2f8375f075fe290a3da2dbceedc8a7d74a94793599cbba1: Status 404 returned error can't find the container with id 34f83950b790ecdac2f8375f075fe290a3da2dbceedc8a7d74a94793599cbba1
Sep 06 10:33:03.479729 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:03.479694    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"]
Sep 06 10:33:03.483366 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:33:03.483313    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3f40fb_76d5_4e58_99b8_e0381ac3e3ea.slice/crio-88ea56e9d99c1d263456cd4a8b3d9dc5c24aaf6ff67cb38d6c31fcee2df07be5 WatchSource:0}: Error finding container 88ea56e9d99c1d263456cd4a8b3d9dc5c24aaf6ff67cb38d6c31fcee2df07be5: Status 404 returned error can't find the container with id 88ea56e9d99c1d263456cd4a8b3d9dc5c24aaf6ff67cb38d6c31fcee2df07be5
Sep 06 10:33:04.209351 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:04.209308    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf" event={"ID":"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea","Type":"ContainerStarted","Data":"fc037f41131a2b8a97bd0223b4ad52eb8d4ab823a8ff811d3488ac4b0ff09e94"}
Sep 06 10:33:04.209833 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:04.209383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf" event={"ID":"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea","Type":"ContainerStarted","Data":"88ea56e9d99c1d263456cd4a8b3d9dc5c24aaf6ff67cb38d6c31fcee2df07be5"}
Sep 06 10:33:04.209833 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:04.209408    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:04.210995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:04.210968    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k" event={"ID":"eb474364-0545-48f7-ad03-76467b42878c","Type":"ContainerStarted","Data":"c2612a3e988cc822ff021ee298fa37a6b985446507cd86efa3ba35f5573965b5"}
Sep 06 10:33:04.211112 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:04.211019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k" event={"ID":"eb474364-0545-48f7-ad03-76467b42878c","Type":"ContainerStarted","Data":"34f83950b790ecdac2f8375f075fe290a3da2dbceedc8a7d74a94793599cbba1"}
Sep 06 10:33:04.211112 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:04.211049    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:04.224991 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:04.224939    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf" podStartSLOduration=2.224919586 podStartE2EDuration="2.224919586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:04.223385315 +0000 UTC m=+675.496026818" watchObservedRunningTime="2026-09-06 10:33:04.224919586 +0000 UTC m=+675.497561091"
Sep 06 10:33:04.238273 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:04.238153    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k" podStartSLOduration=2.238131838 podStartE2EDuration="2.238131838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:04.237810879 +0000 UTC m=+675.510452395" watchObservedRunningTime="2026-09-06 10:33:04.238131838 +0000 UTC m=+675.510773344"
Sep 06 10:33:07.171467 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.171435    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:33:07.175377 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.175347    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:33:07.407500 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.407447    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"]
Sep 06 10:33:07.411010 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.410990    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.413264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.413236    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-5dda3d-serving-cert\""
Sep 06 10:33:07.415015 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.414989    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"]
Sep 06 10:33:07.543815 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.543729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87g6z\" (UniqueName: \"kubernetes.io/projected/9b972f66-b574-455c-b6b2-a4b1abd62b28-kube-api-access-87g6z\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.543958 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.543813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.543958 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.543833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b972f66-b574-455c-b6b2-a4b1abd62b28-openshift-service-ca\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.543958 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.543848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-metrics-certs\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.644715 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.644684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.644715 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.644719    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b972f66-b574-455c-b6b2-a4b1abd62b28-openshift-service-ca\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.644940 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.644743    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-metrics-certs\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.644940 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.644793    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87g6z\" (UniqueName: \"kubernetes.io/projected/9b972f66-b574-455c-b6b2-a4b1abd62b28-kube-api-access-87g6z\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.644940 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:33:07.644899    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-serving-cert: secret "maas-api-e2e-autores-5dda3d-serving-cert" not found
Sep 06 10:33:07.645060 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:33:07.644999    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls podName:9b972f66-b574-455c-b6b2-a4b1abd62b28 nodeName:}" failed. No retries permitted until 2026-09-06 10:33:08.144972933 +0000 UTC m=+679.417614431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls") pod "maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8" (UID: "9b972f66-b574-455c-b6b2-a4b1abd62b28") : secret "maas-api-e2e-autores-5dda3d-serving-cert" not found
Sep 06 10:33:07.645455 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.645434    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b972f66-b574-455c-b6b2-a4b1abd62b28-openshift-service-ca\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.647082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.647064    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-metrics-certs\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:07.652989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:07.652963    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87g6z\" (UniqueName: \"kubernetes.io/projected/9b972f66-b574-455c-b6b2-a4b1abd62b28-kube-api-access-87g6z\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:08.150404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:08.150363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:08.150581 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:33:08.150510    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-serving-cert: secret "maas-api-e2e-autores-5dda3d-serving-cert" not found
Sep 06 10:33:08.150581 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:33:08.150578    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls podName:9b972f66-b574-455c-b6b2-a4b1abd62b28 nodeName:}" failed. No retries permitted until 2026-09-06 10:33:09.150561948 +0000 UTC m=+680.423203447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls") pod "maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8" (UID: "9b972f66-b574-455c-b6b2-a4b1abd62b28") : secret "maas-api-e2e-autores-5dda3d-serving-cert" not found
Sep 06 10:33:09.161153 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:09.161115    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:09.163963 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:09.163935    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls\") pod \"maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:09.180948 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:09.180915    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:33:09.226616 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:09.226548    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:09.507738 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:09.507601    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"]
Sep 06 10:33:09.509964 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:33:09.509937    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b972f66_b574_455c_b6b2_a4b1abd62b28.slice/crio-ba14055629525dcb6e066f981fd0f13173a4d0abab23b4a14631670ed2f4e060 WatchSource:0}: Error finding container ba14055629525dcb6e066f981fd0f13173a4d0abab23b4a14631670ed2f4e060: Status 404 returned error can't find the container with id ba14055629525dcb6e066f981fd0f13173a4d0abab23b4a14631670ed2f4e060
Sep 06 10:33:10.245109 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:10.245018    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8" event={"ID":"9b972f66-b574-455c-b6b2-a4b1abd62b28","Type":"ContainerStarted","Data":"d49cd809c51a231e09993cc6ed2b68a4d4c4c48a07487e0ae049f24c2630f35f"}
Sep 06 10:33:10.245109 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:10.245068    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8" event={"ID":"9b972f66-b574-455c-b6b2-a4b1abd62b28","Type":"ContainerStarted","Data":"ba14055629525dcb6e066f981fd0f13173a4d0abab23b4a14631670ed2f4e060"}
Sep 06 10:33:10.245564 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:10.245113    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:10.261979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:10.261929    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8" podStartSLOduration=2.847929856 podStartE2EDuration="3.26190981s" totalImagesPullingTime="413.979954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:10.260257205 +0000 UTC m=+681.532898722" watchObservedRunningTime="2026-09-06 10:33:10.26190981 +0000 UTC m=+681.534551312"
Sep 06 10:33:15.217138 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:15.217103    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:15.217628 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:15.217176    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:16.255524 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:16.255492    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:33:18.816028 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:18.815987    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:33:24.821831 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:24.821791    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"]
Sep 06 10:33:24.825365 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:24.825343    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:24.827612 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:24.827585    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bd3407\""
Sep 06 10:33:24.827756 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:24.827587    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bd3407-dockercfg-cz4x4\""
Sep 06 10:33:24.834610 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:24.834574    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"]
Sep 06 10:33:24.903328 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:24.903283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds\" (UID: \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:24.903547 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:24.903396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqlk2\" (UniqueName: \"kubernetes.io/projected/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-kube-api-access-rqlk2\") pod \"payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds\" (UID: \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:25.004703 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:25.004654    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds\" (UID: \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:25.004904 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:25.004755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqlk2\" (UniqueName: \"kubernetes.io/projected/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-kube-api-access-rqlk2\") pod \"payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds\" (UID: \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:25.005642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:25.005618    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds\" (UID: \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:25.015213 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:25.015179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqlk2\" (UniqueName: \"kubernetes.io/projected/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-kube-api-access-rqlk2\") pod \"payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds\" (UID: \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:25.136882 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:25.136838    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:25.414177 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:25.414097    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"]
Sep 06 10:33:25.417430 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:33:25.417401    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b75eb94_51f1_4af4_9e26_42ec34d91b6b.slice/crio-20025a9f03911ae5f63ce7a0d24d8202cfa8dad424029f118bca01284006fc1a WatchSource:0}: Error finding container 20025a9f03911ae5f63ce7a0d24d8202cfa8dad424029f118bca01284006fc1a: Status 404 returned error can't find the container with id 20025a9f03911ae5f63ce7a0d24d8202cfa8dad424029f118bca01284006fc1a
Sep 06 10:33:26.323873 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:26.323777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds" event={"ID":"7b75eb94-51f1-4af4-9e26-42ec34d91b6b","Type":"ContainerStarted","Data":"93340deb0476c15b43ad7735490410fa3e48594bd8dc26002a04ad2997ee815a"}
Sep 06 10:33:26.323873 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:26.323828    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds" event={"ID":"7b75eb94-51f1-4af4-9e26-42ec34d91b6b","Type":"ContainerStarted","Data":"20025a9f03911ae5f63ce7a0d24d8202cfa8dad424029f118bca01284006fc1a"}
Sep 06 10:33:26.324415 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:26.323878    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:26.340160 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:26.340103    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds" podStartSLOduration=2.340086005 podStartE2EDuration="2.340086005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:26.338373709 +0000 UTC m=+697.611015213" watchObservedRunningTime="2026-09-06 10:33:26.340086005 +0000 UTC m=+697.612727508"
Sep 06 10:33:34.375575 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:34.375536    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:33:37.331036 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:37.331000    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:33:43.172758 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.172684    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"]
Sep 06 10:33:43.173161 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.172909    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k" podUID="eb474364-0545-48f7-ad03-76467b42878c" containerName="payload-processing" containerID="cri-o://c2612a3e988cc822ff021ee298fa37a6b985446507cd86efa3ba35f5573965b5" gracePeriod=30
Sep 06 10:33:43.178287 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.178257    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"]
Sep 06 10:33:43.178556 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.178507    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf" podUID="3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea" containerName="payload-pre-processing" containerID="cri-o://fc037f41131a2b8a97bd0223b4ad52eb8d4ab823a8ff811d3488ac4b0ff09e94" gracePeriod=30
Sep 06 10:33:43.406637 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.406601    2583 generic.go:356] "Generic (PLEG): container finished" podID="3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea" containerID="fc037f41131a2b8a97bd0223b4ad52eb8d4ab823a8ff811d3488ac4b0ff09e94" exitCode=0
Sep 06 10:33:43.406849 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.406669    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf" event={"ID":"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea","Type":"ContainerDied","Data":"fc037f41131a2b8a97bd0223b4ad52eb8d4ab823a8ff811d3488ac4b0ff09e94"}
Sep 06 10:33:43.408473 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.408442    2583 generic.go:356] "Generic (PLEG): container finished" podID="eb474364-0545-48f7-ad03-76467b42878c" containerID="c2612a3e988cc822ff021ee298fa37a6b985446507cd86efa3ba35f5573965b5" exitCode=0
Sep 06 10:33:43.408590 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.408503    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k" event={"ID":"eb474364-0545-48f7-ad03-76467b42878c","Type":"ContainerDied","Data":"c2612a3e988cc822ff021ee298fa37a6b985446507cd86efa3ba35f5573965b5"}
Sep 06 10:33:43.458762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.458672    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:43.461914 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.461891    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:43.470687 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.470321    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4mm2\" (UniqueName: \"kubernetes.io/projected/eb474364-0545-48f7-ad03-76467b42878c-kube-api-access-k4mm2\") pod \"eb474364-0545-48f7-ad03-76467b42878c\" (UID: \"eb474364-0545-48f7-ad03-76467b42878c\") "
Sep 06 10:33:43.470687 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.470410    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwrkh\" (UniqueName: \"kubernetes.io/projected/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-kube-api-access-fwrkh\") pod \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\" (UID: \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\") "
Sep 06 10:33:43.470687 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.470444    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb474364-0545-48f7-ad03-76467b42878c-plugins-config-volume\") pod \"eb474364-0545-48f7-ad03-76467b42878c\" (UID: \"eb474364-0545-48f7-ad03-76467b42878c\") "
Sep 06 10:33:43.470687 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.470495    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-plugins-config-volume\") pod \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\" (UID: \"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea\") "
Sep 06 10:33:43.471008 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.470797    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb474364-0545-48f7-ad03-76467b42878c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb474364-0545-48f7-ad03-76467b42878c" (UID: "eb474364-0545-48f7-ad03-76467b42878c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:33:43.471164 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.471133    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea" (UID: "3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:33:43.473917 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.473601    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb474364-0545-48f7-ad03-76467b42878c-kube-api-access-k4mm2" (OuterVolumeSpecName: "kube-api-access-k4mm2") pod "eb474364-0545-48f7-ad03-76467b42878c" (UID: "eb474364-0545-48f7-ad03-76467b42878c"). InnerVolumeSpecName "kube-api-access-k4mm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:33:43.473917 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.473688    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-kube-api-access-fwrkh" (OuterVolumeSpecName: "kube-api-access-fwrkh") pod "3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea" (UID: "3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea"). InnerVolumeSpecName "kube-api-access-fwrkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:33:43.571991 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.571950    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4mm2\" (UniqueName: \"kubernetes.io/projected/eb474364-0545-48f7-ad03-76467b42878c-kube-api-access-k4mm2\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:33:43.571991 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.571983    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrkh\" (UniqueName: \"kubernetes.io/projected/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-kube-api-access-fwrkh\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:33:43.571991 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.571992    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb474364-0545-48f7-ad03-76467b42878c-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:33:43.571991 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:43.572001    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:33:44.414591 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.414550    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"
Sep 06 10:33:44.415065 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.414547    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf" event={"ID":"3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea","Type":"ContainerDied","Data":"88ea56e9d99c1d263456cd4a8b3d9dc5c24aaf6ff67cb38d6c31fcee2df07be5"}
Sep 06 10:33:44.415065 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.414722    2583 scope.go:117] "RemoveContainer" containerID="fc037f41131a2b8a97bd0223b4ad52eb8d4ab823a8ff811d3488ac4b0ff09e94"
Sep 06 10:33:44.416391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.416196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k" event={"ID":"eb474364-0545-48f7-ad03-76467b42878c","Type":"ContainerDied","Data":"34f83950b790ecdac2f8375f075fe290a3da2dbceedc8a7d74a94793599cbba1"}
Sep 06 10:33:44.416391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.416211    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"
Sep 06 10:33:44.425915 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.425867    2583 scope.go:117] "RemoveContainer" containerID="c2612a3e988cc822ff021ee298fa37a6b985446507cd86efa3ba35f5573965b5"
Sep 06 10:33:44.454370 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.454304    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"]
Sep 06 10:33:44.461925 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.461896    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dbaf75b1-6b8f95864c-tg76k"]
Sep 06 10:33:44.487856 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.487821    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"]
Sep 06 10:33:44.503269 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:44.503236    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dbaf75b1-5c598cd5cd-njrjf"]
Sep 06 10:33:45.245819 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:45.245787    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea" path="/var/lib/kubelet/pods/3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea/volumes"
Sep 06 10:33:45.246156 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:45.246134    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb474364-0545-48f7-ad03-76467b42878c" path="/var/lib/kubelet/pods/eb474364-0545-48f7-ad03-76467b42878c/volumes"
Sep 06 10:33:47.272104 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.272070    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"]
Sep 06 10:33:47.272627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.272607    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea" containerName="payload-pre-processing"
Sep 06 10:33:47.272753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.272631    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea" containerName="payload-pre-processing"
Sep 06 10:33:47.272753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.272654    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb474364-0545-48f7-ad03-76467b42878c" containerName="payload-processing"
Sep 06 10:33:47.272753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.272660    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb474364-0545-48f7-ad03-76467b42878c" containerName="payload-processing"
Sep 06 10:33:47.272753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.272714    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb474364-0545-48f7-ad03-76467b42878c" containerName="payload-processing"
Sep 06 10:33:47.272753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.272723    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3f40fb-76d5-4e58-99b8-e0381ac3e3ea" containerName="payload-pre-processing"
Sep 06 10:33:47.277276 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.277240    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.279468 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.279442    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ca3c8694-serving-cert\""
Sep 06 10:33:47.291552 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.291525    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"]
Sep 06 10:33:47.300073 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.300045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-maas-api-tls\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.300249 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.300083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-metrics-certs\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.300249 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.300136    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp76r\" (UniqueName: \"kubernetes.io/projected/3b8b5595-4098-482f-a18a-ac79f8451351-kube-api-access-qp76r\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.300249 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.300187    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8b5595-4098-482f-a18a-ac79f8451351-openshift-service-ca\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.400721 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.400683    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-maas-api-tls\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.400721 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.400723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-metrics-certs\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.400948 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.400781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp76r\" (UniqueName: \"kubernetes.io/projected/3b8b5595-4098-482f-a18a-ac79f8451351-kube-api-access-qp76r\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.400948 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.400855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8b5595-4098-482f-a18a-ac79f8451351-openshift-service-ca\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.401596 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.401571    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8b5595-4098-482f-a18a-ac79f8451351-openshift-service-ca\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.403323 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.403302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-metrics-certs\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.403436 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.403381    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-maas-api-tls\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.412704 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.412676    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp76r\" (UniqueName: \"kubernetes.io/projected/3b8b5595-4098-482f-a18a-ac79f8451351-kube-api-access-qp76r\") pod \"maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.587979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.587936    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:47.868775 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:47.868702    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"]
Sep 06 10:33:47.872927 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:33:47.872897    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8b5595_4098_482f_a18a_ac79f8451351.slice/crio-dc9fe8f50abd8da83f4f76df91c4b8cac2c89ec5313fd87a60c53df13ee0a6fc WatchSource:0}: Error finding container dc9fe8f50abd8da83f4f76df91c4b8cac2c89ec5313fd87a60c53df13ee0a6fc: Status 404 returned error can't find the container with id dc9fe8f50abd8da83f4f76df91c4b8cac2c89ec5313fd87a60c53df13ee0a6fc
Sep 06 10:33:48.439075 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:48.439040    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j" event={"ID":"3b8b5595-4098-482f-a18a-ac79f8451351","Type":"ContainerStarted","Data":"a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad"}
Sep 06 10:33:48.439479 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:48.439090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j" event={"ID":"3b8b5595-4098-482f-a18a-ac79f8451351","Type":"ContainerStarted","Data":"dc9fe8f50abd8da83f4f76df91c4b8cac2c89ec5313fd87a60c53df13ee0a6fc"}
Sep 06 10:33:48.439479 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:48.439187    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:33:48.458617 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:48.458566    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j" podStartSLOduration=0.980555131 podStartE2EDuration="1.458546927s" totalImagesPullingTime="477.991796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:48.456900323 +0000 UTC m=+719.729541910" watchObservedRunningTime="2026-09-06 10:33:48.458546927 +0000 UTC m=+719.731188430"
Sep 06 10:33:49.379113 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:49.374408    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:33:54.450106 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:33:54.450074    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:34:05.985095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:05.985053    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:34:16.998179 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:16.998134    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"]
Sep 06 10:34:17.001610 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.001586    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.003750 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.003721    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-30549304-kserve-self-signed-certs\""
Sep 06 10:34:17.015499 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.015470    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"]
Sep 06 10:34:17.061456 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.061417    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-home\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.061614 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.061544    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tmp-dir\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.061667 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.061608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-model-cache\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.061704 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.061689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-dshm\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.061749 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.061721    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tls-certs\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.061790 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.061759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jvl\" (UniqueName: \"kubernetes.io/projected/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-kube-api-access-j2jvl\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.162970 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.162924    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-dshm\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.162970 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.162966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tls-certs\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.163243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.162988    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jvl\" (UniqueName: \"kubernetes.io/projected/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-kube-api-access-j2jvl\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.163308 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.163233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-home\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.163397 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.163323    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tmp-dir\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.163462 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.163445    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-model-cache\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.163625 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.163598    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-home\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.163757 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.163670    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tmp-dir\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.163854 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.163834    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-model-cache\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.165380 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.165328    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-dshm\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.165630 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.165613    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tls-certs\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.171229 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.171202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jvl\" (UniqueName: \"kubernetes.io/projected/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-kube-api-access-j2jvl\") pod \"e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") " pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.311042 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.310955    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:17.586003 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:17.585975    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"]
Sep 06 10:34:17.588145 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:34:17.588114    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8cea9f2_ca2d_41f6_9c64_648d2b7bf21a.slice/crio-4201855d9aae5642cb3c6874d32fd909b8597db81f2dc1e809d3ce5da443e63c WatchSource:0}: Error finding container 4201855d9aae5642cb3c6874d32fd909b8597db81f2dc1e809d3ce5da443e63c: Status 404 returned error can't find the container with id 4201855d9aae5642cb3c6874d32fd909b8597db81f2dc1e809d3ce5da443e63c
Sep 06 10:34:18.586207 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:18.586171    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk" event={"ID":"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a","Type":"ContainerStarted","Data":"66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6"}
Sep 06 10:34:18.586737 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:18.586217    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk" event={"ID":"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a","Type":"ContainerStarted","Data":"4201855d9aae5642cb3c6874d32fd909b8597db81f2dc1e809d3ce5da443e63c"}
Sep 06 10:34:18.605557 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:18.605500    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk" podStartSLOduration=2.6054808019999998 podStartE2EDuration="2.605480802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:34:18.604054701 +0000 UTC m=+749.876696208" watchObservedRunningTime="2026-09-06 10:34:18.605480802 +0000 UTC m=+749.878122305"
Sep 06 10:34:18.779992 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:18.779541    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"]
Sep 06 10:34:18.779992 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:18.779804    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j" podUID="3b8b5595-4098-482f-a18a-ac79f8451351" containerName="maas-api" containerID="cri-o://a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad" gracePeriod=30
Sep 06 10:34:19.034753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.034721    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:34:19.184717 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.184622    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8b5595-4098-482f-a18a-ac79f8451351-openshift-service-ca\") pod \"3b8b5595-4098-482f-a18a-ac79f8451351\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") "
Sep 06 10:34:19.184717 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.184682    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-maas-api-tls\") pod \"3b8b5595-4098-482f-a18a-ac79f8451351\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") "
Sep 06 10:34:19.184941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.184750    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qp76r\" (UniqueName: \"kubernetes.io/projected/3b8b5595-4098-482f-a18a-ac79f8451351-kube-api-access-qp76r\") pod \"3b8b5595-4098-482f-a18a-ac79f8451351\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") "
Sep 06 10:34:19.184941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.184848    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-metrics-certs\") pod \"3b8b5595-4098-482f-a18a-ac79f8451351\" (UID: \"3b8b5595-4098-482f-a18a-ac79f8451351\") "
Sep 06 10:34:19.185055 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.184996    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8b5595-4098-482f-a18a-ac79f8451351-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3b8b5595-4098-482f-a18a-ac79f8451351" (UID: "3b8b5595-4098-482f-a18a-ac79f8451351"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:34:19.185167 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.185147    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b8b5595-4098-482f-a18a-ac79f8451351-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:34:19.188319 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.188270    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3b8b5595-4098-482f-a18a-ac79f8451351" (UID: "3b8b5595-4098-482f-a18a-ac79f8451351"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:34:19.188487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.188314    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3b8b5595-4098-482f-a18a-ac79f8451351" (UID: "3b8b5595-4098-482f-a18a-ac79f8451351"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:34:19.188487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.188399    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8b5595-4098-482f-a18a-ac79f8451351-kube-api-access-qp76r" (OuterVolumeSpecName: "kube-api-access-qp76r") pod "3b8b5595-4098-482f-a18a-ac79f8451351" (UID: "3b8b5595-4098-482f-a18a-ac79f8451351"). InnerVolumeSpecName "kube-api-access-qp76r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:34:19.286187 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.286124    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:34:19.286187 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.286163    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b8b5595-4098-482f-a18a-ac79f8451351-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:34:19.286187 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.286179    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp76r\" (UniqueName: \"kubernetes.io/projected/3b8b5595-4098-482f-a18a-ac79f8451351-kube-api-access-qp76r\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:34:19.592584 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.592550    2583 generic.go:356] "Generic (PLEG): container finished" podID="3b8b5595-4098-482f-a18a-ac79f8451351" containerID="a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad" exitCode=0
Sep 06 10:34:19.593015 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.592627    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"
Sep 06 10:34:19.593015 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.592640    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j" event={"ID":"3b8b5595-4098-482f-a18a-ac79f8451351","Type":"ContainerDied","Data":"a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad"}
Sep 06 10:34:19.593015 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.592693    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j" event={"ID":"3b8b5595-4098-482f-a18a-ac79f8451351","Type":"ContainerDied","Data":"dc9fe8f50abd8da83f4f76df91c4b8cac2c89ec5313fd87a60c53df13ee0a6fc"}
Sep 06 10:34:19.593015 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.592728    2583 scope.go:117] "RemoveContainer" containerID="a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad"
Sep 06 10:34:19.602225 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.602205    2583 scope.go:117] "RemoveContainer" containerID="a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad"
Sep 06 10:34:19.602548 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:34:19.602525    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad\": container with ID starting with a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad not found: ID does not exist" containerID="a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad"
Sep 06 10:34:19.602622 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.602562    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad"} err="failed to get container status \"a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad\": rpc error: code = NotFound desc = could not find container \"a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad\": container with ID starting with a80a914eed4283c0f9205f4514bce9c21713cac7a515fca5fb68b1c30d5e29ad not found: ID does not exist"
Sep 06 10:34:19.608591 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.608566    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"]
Sep 06 10:34:19.614635 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:19.614613    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ca3c8694-7bfd7d78c8-c6p8j"]
Sep 06 10:34:21.240255 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:21.240220    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b8b5595-4098-482f-a18a-ac79f8451351" path="/var/lib/kubelet/pods/3b8b5595-4098-482f-a18a-ac79f8451351/volumes"
Sep 06 10:34:22.374929 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:22.374890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:34:24.326781 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.326740    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"]
Sep 06 10:34:24.327215 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.327197    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b8b5595-4098-482f-a18a-ac79f8451351" containerName="maas-api"
Sep 06 10:34:24.327258 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.327219    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b8b5595-4098-482f-a18a-ac79f8451351" containerName="maas-api"
Sep 06 10:34:24.327314 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.327304    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b8b5595-4098-482f-a18a-ac79f8451351" containerName="maas-api"
Sep 06 10:34:24.337437 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.337396    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:24.338909 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.338880    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"]
Sep 06 10:34:24.339711 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.339688    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-17db6c23\""
Sep 06 10:34:24.339711 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.339705    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-17db6c23-dockercfg-ndrnq\""
Sep 06 10:34:24.434986 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.434943    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsgq9\" (UniqueName: \"kubernetes.io/projected/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-kube-api-access-fsgq9\") pod \"payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv\" (UID: \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:24.435169 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.435027    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv\" (UID: \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:24.536078 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.536039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv\" (UID: \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:24.536283 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.536111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsgq9\" (UniqueName: \"kubernetes.io/projected/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-kube-api-access-fsgq9\") pod \"payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv\" (UID: \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:24.536955 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.536930    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv\" (UID: \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:24.544264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.544233    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsgq9\" (UniqueName: \"kubernetes.io/projected/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-kube-api-access-fsgq9\") pod \"payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv\" (UID: \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:24.650485 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.650442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:24.923933 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:24.923887    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"]
Sep 06 10:34:24.925923 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:34:24.925890    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae179db_f0ce_4dcb_b62b_ebc398bc599d.slice/crio-ac7b1412791dcf0e8b95b4f882989484d45185dddbf3d1e5f233dcd00e2be2fa WatchSource:0}: Error finding container ac7b1412791dcf0e8b95b4f882989484d45185dddbf3d1e5f233dcd00e2be2fa: Status 404 returned error can't find the container with id ac7b1412791dcf0e8b95b4f882989484d45185dddbf3d1e5f233dcd00e2be2fa
Sep 06 10:34:25.626901 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:25.626858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv" event={"ID":"5ae179db-f0ce-4dcb-b62b-ebc398bc599d","Type":"ContainerStarted","Data":"08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3"}
Sep 06 10:34:25.627396 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:25.627024    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:25.627396 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:25.627052    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv" event={"ID":"5ae179db-f0ce-4dcb-b62b-ebc398bc599d","Type":"ContainerStarted","Data":"ac7b1412791dcf0e8b95b4f882989484d45185dddbf3d1e5f233dcd00e2be2fa"}
Sep 06 10:34:25.642275 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:25.642202    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv" podStartSLOduration=1.642180851 podStartE2EDuration="1.642180851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:34:25.6409646 +0000 UTC m=+756.913606106" watchObservedRunningTime="2026-09-06 10:34:25.642180851 +0000 UTC m=+756.914822356"
Sep 06 10:34:27.311661 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:27.311621    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:27.312149 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:27.311684    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:27.324672 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:27.324642    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:27.648901 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:27.648872    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:34:36.634999 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:36.634959    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:39.272100 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:39.272065    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:34:54.407224 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.407179    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"]
Sep 06 10:34:54.407741 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.407487    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv" podUID="5ae179db-f0ce-4dcb-b62b-ebc398bc599d" containerName="payload-pre-processing" containerID="cri-o://08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3" gracePeriod=30
Sep 06 10:34:54.665674 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.665603    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:54.704852 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.704810    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-plugins-config-volume\") pod \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\" (UID: \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\") "
Sep 06 10:34:54.705046 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.704957    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsgq9\" (UniqueName: \"kubernetes.io/projected/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-kube-api-access-fsgq9\") pod \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\" (UID: \"5ae179db-f0ce-4dcb-b62b-ebc398bc599d\") "
Sep 06 10:34:54.705236 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.705215    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ae179db-f0ce-4dcb-b62b-ebc398bc599d" (UID: "5ae179db-f0ce-4dcb-b62b-ebc398bc599d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:34:54.707525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.707494    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-kube-api-access-fsgq9" (OuterVolumeSpecName: "kube-api-access-fsgq9") pod "5ae179db-f0ce-4dcb-b62b-ebc398bc599d" (UID: "5ae179db-f0ce-4dcb-b62b-ebc398bc599d"). InnerVolumeSpecName "kube-api-access-fsgq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:34:54.767449 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.767392    2583 generic.go:356] "Generic (PLEG): container finished" podID="5ae179db-f0ce-4dcb-b62b-ebc398bc599d" containerID="08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3" exitCode=0
Sep 06 10:34:54.767771 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.767529    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"
Sep 06 10:34:54.767908 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.767550    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv" event={"ID":"5ae179db-f0ce-4dcb-b62b-ebc398bc599d","Type":"ContainerDied","Data":"08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3"}
Sep 06 10:34:54.767908 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.767892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv" event={"ID":"5ae179db-f0ce-4dcb-b62b-ebc398bc599d","Type":"ContainerDied","Data":"ac7b1412791dcf0e8b95b4f882989484d45185dddbf3d1e5f233dcd00e2be2fa"}
Sep 06 10:34:54.768023 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.767932    2583 scope.go:117] "RemoveContainer" containerID="08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3"
Sep 06 10:34:54.776884 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.776867    2583 scope.go:117] "RemoveContainer" containerID="08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3"
Sep 06 10:34:54.777221 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:34:54.777202    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3\": container with ID starting with 08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3 not found: ID does not exist" containerID="08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3"
Sep 06 10:34:54.777285 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.777229    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3"} err="failed to get container status \"08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3\": rpc error: code = NotFound desc = could not find container \"08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3\": container with ID starting with 08ad390ca9aab33824b87c952ad291062c564abff63f61288bfa7c71945945b3 not found: ID does not exist"
Sep 06 10:34:54.790082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.790050    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"]
Sep 06 10:34:54.793857 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.793827    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-17db6c23-6b97c84474-5x7bv"]
Sep 06 10:34:54.806678 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.806645    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsgq9\" (UniqueName: \"kubernetes.io/projected/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-kube-api-access-fsgq9\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:34:54.806678 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:54.806674    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae179db-f0ce-4dcb-b62b-ebc398bc599d-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:34:55.239601 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:55.239568    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae179db-f0ce-4dcb-b62b-ebc398bc599d" path="/var/lib/kubelet/pods/5ae179db-f0ce-4dcb-b62b-ebc398bc599d/volumes"
Sep 06 10:34:55.380006 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:34:55.379970    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:35:00.632893 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.632860    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"]
Sep 06 10:35:00.633404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.633322    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae179db-f0ce-4dcb-b62b-ebc398bc599d" containerName="payload-pre-processing"
Sep 06 10:35:00.633404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.633374    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae179db-f0ce-4dcb-b62b-ebc398bc599d" containerName="payload-pre-processing"
Sep 06 10:35:00.633526 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.633471    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae179db-f0ce-4dcb-b62b-ebc398bc599d" containerName="payload-pre-processing"
Sep 06 10:35:00.638016 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.637996    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:00.640372 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.640353    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-7bb9997d-dockercfg-jzdxm\""
Sep 06 10:35:00.640482 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.640453    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-7bb9997d\""
Sep 06 10:35:00.646859 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.646831    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"]
Sep 06 10:35:00.761443 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.761398    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87w9g\" (UniqueName: \"kubernetes.io/projected/f2988c1c-4f59-40bc-8da6-788c30d1f440-kube-api-access-87w9g\") pod \"payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx\" (UID: \"f2988c1c-4f59-40bc-8da6-788c30d1f440\") " pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:00.761625 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.761512    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2988c1c-4f59-40bc-8da6-788c30d1f440-plugins-config-volume\") pod \"payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx\" (UID: \"f2988c1c-4f59-40bc-8da6-788c30d1f440\") " pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:00.862844 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.862798    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87w9g\" (UniqueName: \"kubernetes.io/projected/f2988c1c-4f59-40bc-8da6-788c30d1f440-kube-api-access-87w9g\") pod \"payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx\" (UID: \"f2988c1c-4f59-40bc-8da6-788c30d1f440\") " pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:00.863023 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.862865    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2988c1c-4f59-40bc-8da6-788c30d1f440-plugins-config-volume\") pod \"payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx\" (UID: \"f2988c1c-4f59-40bc-8da6-788c30d1f440\") " pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:00.863596 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.863577    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2988c1c-4f59-40bc-8da6-788c30d1f440-plugins-config-volume\") pod \"payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx\" (UID: \"f2988c1c-4f59-40bc-8da6-788c30d1f440\") " pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:00.871220 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.871186    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87w9g\" (UniqueName: \"kubernetes.io/projected/f2988c1c-4f59-40bc-8da6-788c30d1f440-kube-api-access-87w9g\") pod \"payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx\" (UID: \"f2988c1c-4f59-40bc-8da6-788c30d1f440\") " pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:00.949528 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:00.949434    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:01.224775 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:01.224696    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"]
Sep 06 10:35:01.228755 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:35:01.228718    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2988c1c_4f59_40bc_8da6_788c30d1f440.slice/crio-5379f58269264ba0ab202370a77acdcbffb10fb071452714d8843f9cabbba515 WatchSource:0}: Error finding container 5379f58269264ba0ab202370a77acdcbffb10fb071452714d8843f9cabbba515: Status 404 returned error can't find the container with id 5379f58269264ba0ab202370a77acdcbffb10fb071452714d8843f9cabbba515
Sep 06 10:35:01.804083 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:01.804051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx" event={"ID":"f2988c1c-4f59-40bc-8da6-788c30d1f440","Type":"ContainerStarted","Data":"b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8"}
Sep 06 10:35:01.804083 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:01.804101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx" event={"ID":"f2988c1c-4f59-40bc-8da6-788c30d1f440","Type":"ContainerStarted","Data":"5379f58269264ba0ab202370a77acdcbffb10fb071452714d8843f9cabbba515"}
Sep 06 10:35:01.804644 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:01.804183    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:01.823194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:01.823138    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx" podStartSLOduration=1.823120793 podStartE2EDuration="1.823120793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:35:01.822032182 +0000 UTC m=+793.094673700" watchObservedRunningTime="2026-09-06 10:35:01.823120793 +0000 UTC m=+793.095762296"
Sep 06 10:35:12.811404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:12.811309    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:16.274893 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:16.274853    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:35:20.089638 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:20.089596    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"]
Sep 06 10:35:30.472992 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.472952    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"]
Sep 06 10:35:30.473523 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.473213    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx" podUID="f2988c1c-4f59-40bc-8da6-788c30d1f440" containerName="payload-processing" containerID="cri-o://b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8" gracePeriod=30
Sep 06 10:35:30.734149 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.734124    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:30.823569 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.823535    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2988c1c-4f59-40bc-8da6-788c30d1f440-plugins-config-volume\") pod \"f2988c1c-4f59-40bc-8da6-788c30d1f440\" (UID: \"f2988c1c-4f59-40bc-8da6-788c30d1f440\") "
Sep 06 10:35:30.823740 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.823615    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87w9g\" (UniqueName: \"kubernetes.io/projected/f2988c1c-4f59-40bc-8da6-788c30d1f440-kube-api-access-87w9g\") pod \"f2988c1c-4f59-40bc-8da6-788c30d1f440\" (UID: \"f2988c1c-4f59-40bc-8da6-788c30d1f440\") "
Sep 06 10:35:30.823971 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.823946    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2988c1c-4f59-40bc-8da6-788c30d1f440-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2988c1c-4f59-40bc-8da6-788c30d1f440" (UID: "f2988c1c-4f59-40bc-8da6-788c30d1f440"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:35:30.825709 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.825677    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2988c1c-4f59-40bc-8da6-788c30d1f440-kube-api-access-87w9g" (OuterVolumeSpecName: "kube-api-access-87w9g") pod "f2988c1c-4f59-40bc-8da6-788c30d1f440" (UID: "f2988c1c-4f59-40bc-8da6-788c30d1f440"). InnerVolumeSpecName "kube-api-access-87w9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:30.924547 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.924516    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2988c1c-4f59-40bc-8da6-788c30d1f440-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:30.924547 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.924543    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87w9g\" (UniqueName: \"kubernetes.io/projected/f2988c1c-4f59-40bc-8da6-788c30d1f440-kube-api-access-87w9g\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:30.942660 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.942628    2583 generic.go:356] "Generic (PLEG): container finished" podID="f2988c1c-4f59-40bc-8da6-788c30d1f440" containerID="b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8" exitCode=0
Sep 06 10:35:30.942821 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.942668    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx" event={"ID":"f2988c1c-4f59-40bc-8da6-788c30d1f440","Type":"ContainerDied","Data":"b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8"}
Sep 06 10:35:30.942821 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.942673    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"
Sep 06 10:35:30.942821 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.942708    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx" event={"ID":"f2988c1c-4f59-40bc-8da6-788c30d1f440","Type":"ContainerDied","Data":"5379f58269264ba0ab202370a77acdcbffb10fb071452714d8843f9cabbba515"}
Sep 06 10:35:30.942821 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.942729    2583 scope.go:117] "RemoveContainer" containerID="b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8"
Sep 06 10:35:30.951458 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.951433    2583 scope.go:117] "RemoveContainer" containerID="b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8"
Sep 06 10:35:30.951752 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:35:30.951731    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8\": container with ID starting with b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8 not found: ID does not exist" containerID="b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8"
Sep 06 10:35:30.951796 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.951762    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8"} err="failed to get container status \"b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8\": rpc error: code = NotFound desc = could not find container \"b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8\": container with ID starting with b270e53c771c8f9e1fe692bccb41104f5dd889628d821e15b047b9611562acd8 not found: ID does not exist"
Sep 06 10:35:30.963067 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.963033    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"]
Sep 06 10:35:30.967117 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:30.967093    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-7bb9997d-76dc57cd9c-lhmqx"]
Sep 06 10:35:31.239896 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:31.239864    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2988c1c-4f59-40bc-8da6-788c30d1f440" path="/var/lib/kubelet/pods/f2988c1c-4f59-40bc-8da6-788c30d1f440/volumes"
Sep 06 10:35:32.841771 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:32.841735    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"]
Sep 06 10:35:32.842243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:32.842042    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8" podUID="9b972f66-b574-455c-b6b2-a4b1abd62b28" containerName="maas-api" containerID="cri-o://d49cd809c51a231e09993cc6ed2b68a4d4c4c48a07487e0ae049f24c2630f35f" gracePeriod=30
Sep 06 10:35:32.957839 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:32.957793    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b972f66-b574-455c-b6b2-a4b1abd62b28" containerID="d49cd809c51a231e09993cc6ed2b68a4d4c4c48a07487e0ae049f24c2630f35f" exitCode=0
Sep 06 10:35:32.957980 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:32.957835    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8" event={"ID":"9b972f66-b574-455c-b6b2-a4b1abd62b28","Type":"ContainerDied","Data":"d49cd809c51a231e09993cc6ed2b68a4d4c4c48a07487e0ae049f24c2630f35f"}
Sep 06 10:35:33.094723 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.094661    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:35:33.145006 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.144971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-metrics-certs\") pod \"9b972f66-b574-455c-b6b2-a4b1abd62b28\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") "
Sep 06 10:35:33.145198 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.145018    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b972f66-b574-455c-b6b2-a4b1abd62b28-openshift-service-ca\") pod \"9b972f66-b574-455c-b6b2-a4b1abd62b28\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") "
Sep 06 10:35:33.145198 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.145099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87g6z\" (UniqueName: \"kubernetes.io/projected/9b972f66-b574-455c-b6b2-a4b1abd62b28-kube-api-access-87g6z\") pod \"9b972f66-b574-455c-b6b2-a4b1abd62b28\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") "
Sep 06 10:35:33.145198 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.145120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls\") pod \"9b972f66-b574-455c-b6b2-a4b1abd62b28\" (UID: \"9b972f66-b574-455c-b6b2-a4b1abd62b28\") "
Sep 06 10:35:33.145466 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.145441    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b972f66-b574-455c-b6b2-a4b1abd62b28-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b972f66-b574-455c-b6b2-a4b1abd62b28" (UID: "9b972f66-b574-455c-b6b2-a4b1abd62b28"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:35:33.147250 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.147217    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b972f66-b574-455c-b6b2-a4b1abd62b28-kube-api-access-87g6z" (OuterVolumeSpecName: "kube-api-access-87g6z") pod "9b972f66-b574-455c-b6b2-a4b1abd62b28" (UID: "9b972f66-b574-455c-b6b2-a4b1abd62b28"). InnerVolumeSpecName "kube-api-access-87g6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:33.147250 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.147233    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b972f66-b574-455c-b6b2-a4b1abd62b28" (UID: "9b972f66-b574-455c-b6b2-a4b1abd62b28"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:35:33.147431 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.147272    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b972f66-b574-455c-b6b2-a4b1abd62b28" (UID: "9b972f66-b574-455c-b6b2-a4b1abd62b28"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:35:33.246797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.246732    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:33.246797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.246767    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b972f66-b574-455c-b6b2-a4b1abd62b28-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:33.246797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.246785    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87g6z\" (UniqueName: \"kubernetes.io/projected/9b972f66-b574-455c-b6b2-a4b1abd62b28-kube-api-access-87g6z\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:33.246797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.246801    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b972f66-b574-455c-b6b2-a4b1abd62b28-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:33.771935 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.771895    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:35:33.964776 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.964746    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"
Sep 06 10:35:33.965256 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.964786    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8" event={"ID":"9b972f66-b574-455c-b6b2-a4b1abd62b28","Type":"ContainerDied","Data":"ba14055629525dcb6e066f981fd0f13173a4d0abab23b4a14631670ed2f4e060"}
Sep 06 10:35:33.965256 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.964846    2583 scope.go:117] "RemoveContainer" containerID="d49cd809c51a231e09993cc6ed2b68a4d4c4c48a07487e0ae049f24c2630f35f"
Sep 06 10:35:33.981136 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.981109    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"]
Sep 06 10:35:33.985399 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:33.985375    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5dda3d-88f4dc686-4hkb8"]
Sep 06 10:35:35.110857 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.110796    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk" podUID="c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" containerName="main" containerID="cri-o://66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6" gracePeriod=45
Sep 06 10:35:35.242315 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.242282    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b972f66-b574-455c-b6b2-a4b1abd62b28" path="/var/lib/kubelet/pods/9b972f66-b574-455c-b6b2-a4b1abd62b28/volumes"
Sep 06 10:35:35.365171 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.365101    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:35:35.467066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467035    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2jvl\" (UniqueName: \"kubernetes.io/projected/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-kube-api-access-j2jvl\") pod \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") "
Sep 06 10:35:35.467229 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467101    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tmp-dir\") pod \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") "
Sep 06 10:35:35.467229 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467120    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-home\") pod \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") "
Sep 06 10:35:35.467229 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tls-certs\") pod \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") "
Sep 06 10:35:35.467431 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467300    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-dshm\") pod \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") "
Sep 06 10:35:35.467431 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467359    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-home" (OuterVolumeSpecName: "home") pod "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" (UID: "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:35.467431 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467373    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" (UID: "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:35.467431 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467375    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-model-cache\") pod \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\" (UID: \"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a\") "
Sep 06 10:35:35.467624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467562    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" (UID: "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:35.467675 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467659    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tmp-dir\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:35.467713 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467681    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-home\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:35.467713 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.467693    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-model-cache\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:35.469485 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.469448    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-kube-api-access-j2jvl" (OuterVolumeSpecName: "kube-api-access-j2jvl") pod "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" (UID: "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a"). InnerVolumeSpecName "kube-api-access-j2jvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:35:35.469817 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.469793    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-dshm" (OuterVolumeSpecName: "dshm") pod "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" (UID: "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:35:35.469817 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.469809    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" (UID: "c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:35:35.568266 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.568231    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-dshm\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:35.568266 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.568260    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jvl\" (UniqueName: \"kubernetes.io/projected/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-kube-api-access-j2jvl\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:35.568266 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.568272    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a-tls-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:35:35.977633 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.977602    2583 generic.go:356] "Generic (PLEG): container finished" podID="c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" containerID="66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6" exitCode=0
Sep 06 10:35:35.977841 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.977646    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"
Sep 06 10:35:35.977841 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.977695    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk" event={"ID":"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a","Type":"ContainerDied","Data":"66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6"}
Sep 06 10:35:35.977841 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.977744    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk" event={"ID":"c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a","Type":"ContainerDied","Data":"4201855d9aae5642cb3c6874d32fd909b8597db81f2dc1e809d3ce5da443e63c"}
Sep 06 10:35:35.977841 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.977766    2583 scope.go:117] "RemoveContainer" containerID="66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6"
Sep 06 10:35:35.991957 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.991919    2583 scope.go:117] "RemoveContainer" containerID="66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6"
Sep 06 10:35:35.992560 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:35:35.992527    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6\": container with ID starting with 66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6 not found: ID does not exist" containerID="66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6"
Sep 06 10:35:35.992662 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:35.992574    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6"} err="failed to get container status \"66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6\": rpc error: code = NotFound desc = could not find container \"66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6\": container with ID starting with 66d78c2109327ab5e9d9216d3504cf3526c99f4d8aeeefafbbd3eedfdc6115c6 not found: ID does not exist"
Sep 06 10:35:36.002085 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:36.002056    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"]
Sep 06 10:35:36.003574 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:36.003552    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-30549304-kserve-6856688f7c-8xnwk"]
Sep 06 10:35:37.240321 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:37.240286    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" path="/var/lib/kubelet/pods/c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a/volumes"
Sep 06 10:35:45.216325 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.216279    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"]
Sep 06 10:35:45.216947 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.216920    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" containerName="main"
Sep 06 10:35:45.217034 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.216951    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" containerName="main"
Sep 06 10:35:45.217034 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.216975    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2988c1c-4f59-40bc-8da6-788c30d1f440" containerName="payload-processing"
Sep 06 10:35:45.217034 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.216984    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2988c1c-4f59-40bc-8da6-788c30d1f440" containerName="payload-processing"
Sep 06 10:35:45.217034 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.217005    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b972f66-b574-455c-b6b2-a4b1abd62b28" containerName="maas-api"
Sep 06 10:35:45.217034 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.217014    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b972f66-b574-455c-b6b2-a4b1abd62b28" containerName="maas-api"
Sep 06 10:35:45.217207 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.217110    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8cea9f2-ca2d-41f6-9c64-648d2b7bf21a" containerName="main"
Sep 06 10:35:45.217207 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.217121    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b972f66-b574-455c-b6b2-a4b1abd62b28" containerName="maas-api"
Sep 06 10:35:45.217207 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.217133    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2988c1c-4f59-40bc-8da6-788c30d1f440" containerName="payload-processing"
Sep 06 10:35:45.220600 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.220575    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.222754 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.222732    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-6c8f35-serving-cert\""
Sep 06 10:35:45.227807 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.227781    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"]
Sep 06 10:35:45.262637 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.262599    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-metrics-certs\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.262830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.262663    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67930c13-3202-431a-9968-ee3f7bd6ad42-openshift-service-ca\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.262830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.262750    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-maas-api-tls\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.262830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.262774    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6bfv\" (UniqueName: \"kubernetes.io/projected/67930c13-3202-431a-9968-ee3f7bd6ad42-kube-api-access-j6bfv\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.364130 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.364087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-maas-api-tls\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.364130 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.364131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6bfv\" (UniqueName: \"kubernetes.io/projected/67930c13-3202-431a-9968-ee3f7bd6ad42-kube-api-access-j6bfv\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.364428 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.364187    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-metrics-certs\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.364428 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.364252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67930c13-3202-431a-9968-ee3f7bd6ad42-openshift-service-ca\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.364878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.364852    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67930c13-3202-431a-9968-ee3f7bd6ad42-openshift-service-ca\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.366557 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.366537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-maas-api-tls\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.366643 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.366625    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-metrics-certs\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.371501 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.371472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6bfv\" (UniqueName: \"kubernetes.io/projected/67930c13-3202-431a-9968-ee3f7bd6ad42-kube-api-access-j6bfv\") pod \"maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.535487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.535392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:45.810044 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:45.809966    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"]
Sep 06 10:35:45.813884 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:35:45.813852    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67930c13_3202_431a_9968_ee3f7bd6ad42.slice/crio-14936f95460306ef1a111831324b9600e7983f9f1c7d4a0c5a1a8acc724e5591 WatchSource:0}: Error finding container 14936f95460306ef1a111831324b9600e7983f9f1c7d4a0c5a1a8acc724e5591: Status 404 returned error can't find the container with id 14936f95460306ef1a111831324b9600e7983f9f1c7d4a0c5a1a8acc724e5591
Sep 06 10:35:46.032197 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:46.029902    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt" event={"ID":"67930c13-3202-431a-9968-ee3f7bd6ad42","Type":"ContainerStarted","Data":"14936f95460306ef1a111831324b9600e7983f9f1c7d4a0c5a1a8acc724e5591"}
Sep 06 10:35:47.035907 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:47.035873    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt" event={"ID":"67930c13-3202-431a-9968-ee3f7bd6ad42","Type":"ContainerStarted","Data":"e16ad39fcd3ed149d5705e81c7476bacef177f5332052aa3ca5bf3d8caeb9ad9"}
Sep 06 10:35:47.036352 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:47.035981    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:35:47.052521 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:47.052471    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt" podStartSLOduration=1.52538818 podStartE2EDuration="2.052454292s" totalImagesPullingTime="527.066112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:35:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:35:47.051359831 +0000 UTC m=+838.324001328" watchObservedRunningTime="2026-09-06 10:35:47.052454292 +0000 UTC m=+838.325095794"
Sep 06 10:35:50.882260 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:50.882219    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:35:53.045798 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:35:53.045770    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:36:06.372099 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:06.372064    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:36:20.880078 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:20.880036    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:36:38.973347 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:38.973292    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:36:50.779890 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:50.779852    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:36:52.178534 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.178499    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"]
Sep 06 10:36:52.179031 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.178769    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt" podUID="67930c13-3202-431a-9968-ee3f7bd6ad42" containerName="maas-api" containerID="cri-o://e16ad39fcd3ed149d5705e81c7476bacef177f5332052aa3ca5bf3d8caeb9ad9" gracePeriod=30
Sep 06 10:36:52.342508 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.342470    2583 generic.go:356] "Generic (PLEG): container finished" podID="67930c13-3202-431a-9968-ee3f7bd6ad42" containerID="e16ad39fcd3ed149d5705e81c7476bacef177f5332052aa3ca5bf3d8caeb9ad9" exitCode=0
Sep 06 10:36:52.342676 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.342543    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt" event={"ID":"67930c13-3202-431a-9968-ee3f7bd6ad42","Type":"ContainerDied","Data":"e16ad39fcd3ed149d5705e81c7476bacef177f5332052aa3ca5bf3d8caeb9ad9"}
Sep 06 10:36:52.435689 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.435622    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:36:52.487349 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.487307    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67930c13-3202-431a-9968-ee3f7bd6ad42-openshift-service-ca\") pod \"67930c13-3202-431a-9968-ee3f7bd6ad42\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") "
Sep 06 10:36:52.487503 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.487361    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-metrics-certs\") pod \"67930c13-3202-431a-9968-ee3f7bd6ad42\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") "
Sep 06 10:36:52.487503 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.487440    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-maas-api-tls\") pod \"67930c13-3202-431a-9968-ee3f7bd6ad42\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") "
Sep 06 10:36:52.487581 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.487508    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6bfv\" (UniqueName: \"kubernetes.io/projected/67930c13-3202-431a-9968-ee3f7bd6ad42-kube-api-access-j6bfv\") pod \"67930c13-3202-431a-9968-ee3f7bd6ad42\" (UID: \"67930c13-3202-431a-9968-ee3f7bd6ad42\") "
Sep 06 10:36:52.487687 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.487666    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67930c13-3202-431a-9968-ee3f7bd6ad42-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67930c13-3202-431a-9968-ee3f7bd6ad42" (UID: "67930c13-3202-431a-9968-ee3f7bd6ad42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:36:52.487832 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.487814    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67930c13-3202-431a-9968-ee3f7bd6ad42-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:36:52.489500 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.489478    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67930c13-3202-431a-9968-ee3f7bd6ad42" (UID: "67930c13-3202-431a-9968-ee3f7bd6ad42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:36:52.489584 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.489543    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67930c13-3202-431a-9968-ee3f7bd6ad42-kube-api-access-j6bfv" (OuterVolumeSpecName: "kube-api-access-j6bfv") pod "67930c13-3202-431a-9968-ee3f7bd6ad42" (UID: "67930c13-3202-431a-9968-ee3f7bd6ad42"). InnerVolumeSpecName "kube-api-access-j6bfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:36:52.489584 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.489572    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "67930c13-3202-431a-9968-ee3f7bd6ad42" (UID: "67930c13-3202-431a-9968-ee3f7bd6ad42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:36:52.588355 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.588311    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:36:52.588519 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.588372    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6bfv\" (UniqueName: \"kubernetes.io/projected/67930c13-3202-431a-9968-ee3f7bd6ad42-kube-api-access-j6bfv\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:36:52.588519 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:52.588385    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67930c13-3202-431a-9968-ee3f7bd6ad42-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:36:53.348555 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:53.348521    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"
Sep 06 10:36:53.348978 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:53.348521    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt" event={"ID":"67930c13-3202-431a-9968-ee3f7bd6ad42","Type":"ContainerDied","Data":"14936f95460306ef1a111831324b9600e7983f9f1c7d4a0c5a1a8acc724e5591"}
Sep 06 10:36:53.348978 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:53.348676    2583 scope.go:117] "RemoveContainer" containerID="e16ad39fcd3ed149d5705e81c7476bacef177f5332052aa3ca5bf3d8caeb9ad9"
Sep 06 10:36:53.363771 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:53.363744    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"]
Sep 06 10:36:53.372769 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:53.372738    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-6c8f35-5f7d9cf79-lgkvt"]
Sep 06 10:36:55.241910 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:36:55.241874    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67930c13-3202-431a-9968-ee3f7bd6ad42" path="/var/lib/kubelet/pods/67930c13-3202-431a-9968-ee3f7bd6ad42/volumes"
Sep 06 10:37:04.967992 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:04.967950    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:37:12.389616 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.389583    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"]
Sep 06 10:37:12.390203 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.389968    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67930c13-3202-431a-9968-ee3f7bd6ad42" containerName="maas-api"
Sep 06 10:37:12.390203 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.389982    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="67930c13-3202-431a-9968-ee3f7bd6ad42" containerName="maas-api"
Sep 06 10:37:12.390203 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.390051    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="67930c13-3202-431a-9968-ee3f7bd6ad42" containerName="maas-api"
Sep 06 10:37:12.394679 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.394654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.397062 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.397034    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f3881d63-serving-cert\""
Sep 06 10:37:12.406299 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.406267    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"]
Sep 06 10:37:12.448138 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.448103    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-maas-api-tls\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.448138 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.448145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-metrics-certs\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.448420 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.448169    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj7lm\" (UniqueName: \"kubernetes.io/projected/67da3e2d-13d8-4aa1-8b27-f444145d31ae-kube-api-access-zj7lm\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.448420 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.448288    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67da3e2d-13d8-4aa1-8b27-f444145d31ae-openshift-service-ca\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.548921 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.548878    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj7lm\" (UniqueName: \"kubernetes.io/projected/67da3e2d-13d8-4aa1-8b27-f444145d31ae-kube-api-access-zj7lm\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.548921 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.548928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67da3e2d-13d8-4aa1-8b27-f444145d31ae-openshift-service-ca\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.549187 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.549005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-maas-api-tls\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.549187 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.549021    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-metrics-certs\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.549187 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:37:12.549107    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-serving-cert: secret "maas-api-e2e-mt-f3881d63-serving-cert" not found
Sep 06 10:37:12.549187 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:37:12.549179    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-maas-api-tls podName:67da3e2d-13d8-4aa1-8b27-f444145d31ae nodeName:}" failed. No retries permitted until 2026-09-06 10:37:13.049162395 +0000 UTC m=+924.321803896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-maas-api-tls") pod "maas-api-e2e-mt-f3881d63-6574df6754-mxld5" (UID: "67da3e2d-13d8-4aa1-8b27-f444145d31ae") : secret "maas-api-e2e-mt-f3881d63-serving-cert" not found
Sep 06 10:37:12.549662 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.549641    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67da3e2d-13d8-4aa1-8b27-f444145d31ae-openshift-service-ca\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.551383 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.551358    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-metrics-certs\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:12.560081 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:12.560054    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj7lm\" (UniqueName: \"kubernetes.io/projected/67da3e2d-13d8-4aa1-8b27-f444145d31ae-kube-api-access-zj7lm\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:13.054250 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:13.054217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-maas-api-tls\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:13.056731 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:13.056699    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-maas-api-tls\") pod \"maas-api-e2e-mt-f3881d63-6574df6754-mxld5\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:13.306377 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:13.306244    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:13.585930 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:13.585900    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"]
Sep 06 10:37:13.588629 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:37:13.588598    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67da3e2d_13d8_4aa1_8b27_f444145d31ae.slice/crio-610e704059bbc2be661a49b81a3cab578b1e3ae9c71bec9f7adcf215ead29f45 WatchSource:0}: Error finding container 610e704059bbc2be661a49b81a3cab578b1e3ae9c71bec9f7adcf215ead29f45: Status 404 returned error can't find the container with id 610e704059bbc2be661a49b81a3cab578b1e3ae9c71bec9f7adcf215ead29f45
Sep 06 10:37:14.452866 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:14.452831    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5" event={"ID":"67da3e2d-13d8-4aa1-8b27-f444145d31ae","Type":"ContainerStarted","Data":"7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429"}
Sep 06 10:37:14.453058 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:14.452892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5" event={"ID":"67da3e2d-13d8-4aa1-8b27-f444145d31ae","Type":"ContainerStarted","Data":"610e704059bbc2be661a49b81a3cab578b1e3ae9c71bec9f7adcf215ead29f45"}
Sep 06 10:37:14.453058 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:14.452973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:14.469661 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:14.469612    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5" podStartSLOduration=2.086160682 podStartE2EDuration="2.469589884s" totalImagesPullingTime="383.429202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:14.468504849 +0000 UTC m=+925.741146365" watchObservedRunningTime="2026-09-06 10:37:14.469589884 +0000 UTC m=+925.742231436"
Sep 06 10:37:20.372403 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:20.372366    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:37:20.463002 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:20.462971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:41.268504 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:41.268468    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:37:53.969200 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:53.969154    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:37:54.230704 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.230604    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"]
Sep 06 10:37:54.230899 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.230866    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5" podUID="67da3e2d-13d8-4aa1-8b27-f444145d31ae" containerName="maas-api" containerID="cri-o://7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429" gracePeriod=30
Sep 06 10:37:54.486060 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.486000    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:54.521178 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.521146    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67da3e2d-13d8-4aa1-8b27-f444145d31ae-openshift-service-ca\") pod \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") "
Sep 06 10:37:54.521395 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.521210    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj7lm\" (UniqueName: \"kubernetes.io/projected/67da3e2d-13d8-4aa1-8b27-f444145d31ae-kube-api-access-zj7lm\") pod \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") "
Sep 06 10:37:54.521395 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.521237    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-metrics-certs\") pod \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") "
Sep 06 10:37:54.521395 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.521263    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-maas-api-tls\") pod \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\" (UID: \"67da3e2d-13d8-4aa1-8b27-f444145d31ae\") "
Sep 06 10:37:54.521652 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.521622    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67da3e2d-13d8-4aa1-8b27-f444145d31ae-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67da3e2d-13d8-4aa1-8b27-f444145d31ae" (UID: "67da3e2d-13d8-4aa1-8b27-f444145d31ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:37:54.524130 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.524095    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67da3e2d-13d8-4aa1-8b27-f444145d31ae-kube-api-access-zj7lm" (OuterVolumeSpecName: "kube-api-access-zj7lm") pod "67da3e2d-13d8-4aa1-8b27-f444145d31ae" (UID: "67da3e2d-13d8-4aa1-8b27-f444145d31ae"). InnerVolumeSpecName "kube-api-access-zj7lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:54.524258 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.524098    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67da3e2d-13d8-4aa1-8b27-f444145d31ae" (UID: "67da3e2d-13d8-4aa1-8b27-f444145d31ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:37:54.524258 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.524218    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "67da3e2d-13d8-4aa1-8b27-f444145d31ae" (UID: "67da3e2d-13d8-4aa1-8b27-f444145d31ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:37:54.622242 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.622204    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67da3e2d-13d8-4aa1-8b27-f444145d31ae-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:54.622242 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.622233    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj7lm\" (UniqueName: \"kubernetes.io/projected/67da3e2d-13d8-4aa1-8b27-f444145d31ae-kube-api-access-zj7lm\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:54.622242 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.622244    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:54.622481 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.622252    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67da3e2d-13d8-4aa1-8b27-f444145d31ae-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:54.642967 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.642929    2583 generic.go:356] "Generic (PLEG): container finished" podID="67da3e2d-13d8-4aa1-8b27-f444145d31ae" containerID="7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429" exitCode=0
Sep 06 10:37:54.643186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.642994    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"
Sep 06 10:37:54.643186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.643037    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5" event={"ID":"67da3e2d-13d8-4aa1-8b27-f444145d31ae","Type":"ContainerDied","Data":"7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429"}
Sep 06 10:37:54.643186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.643095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5" event={"ID":"67da3e2d-13d8-4aa1-8b27-f444145d31ae","Type":"ContainerDied","Data":"610e704059bbc2be661a49b81a3cab578b1e3ae9c71bec9f7adcf215ead29f45"}
Sep 06 10:37:54.643186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.643122    2583 scope.go:117] "RemoveContainer" containerID="7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429"
Sep 06 10:37:54.654355 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.654304    2583 scope.go:117] "RemoveContainer" containerID="7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429"
Sep 06 10:37:54.654695 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:37:54.654671    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429\": container with ID starting with 7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429 not found: ID does not exist" containerID="7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429"
Sep 06 10:37:54.654762 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.654705    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429"} err="failed to get container status \"7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429\": rpc error: code = NotFound desc = could not find container \"7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429\": container with ID starting with 7c55e45515db73f9f592a4aaa556f6760868b6e06e447e0af2dd57b76fdd6429 not found: ID does not exist"
Sep 06 10:37:54.663965 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.663938    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"]
Sep 06 10:37:54.669221 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:54.669187    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3881d63-6574df6754-mxld5"]
Sep 06 10:37:55.241013 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:55.240981    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67da3e2d-13d8-4aa1-8b27-f444145d31ae" path="/var/lib/kubelet/pods/67da3e2d-13d8-4aa1-8b27-f444145d31ae/volumes"
Sep 06 10:37:56.865919 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:56.865876    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"]
Sep 06 10:37:56.866306 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:56.866162    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm" podUID="27d6fbc2-2eb7-4403-973b-6b63ef7d188b" containerName="maas-api" containerID="cri-o://bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6" gracePeriod=30
Sep 06 10:37:56.916852 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:56.916820    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"]
Sep 06 10:37:56.917130 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:56.917074    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227" podUID="3725842c-df7c-4b74-b6e5-6103c49f8675" containerName="payload-processing" containerID="cri-o://de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d" gracePeriod=30
Sep 06 10:37:56.922642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:56.922615    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"]
Sep 06 10:37:56.923527 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:56.922984    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g" podUID="f4c88fab-1746-46eb-b51a-1b8d1dd4e485" containerName="payload-pre-processing" containerID="cri-o://57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d" gracePeriod=30
Sep 06 10:37:57.180754 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.180468    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:37:57.198576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.198547    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:37:57.226571 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.226535    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:37:57.243690 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.243637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-maas-api-tls\") pod \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") "
Sep 06 10:37:57.244112 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.244085    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-plugins-config-volume\") pod \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\" (UID: \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\") "
Sep 06 10:37:57.244230 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.244171    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3725842c-df7c-4b74-b6e5-6103c49f8675-plugins-config-volume\") pod \"3725842c-df7c-4b74-b6e5-6103c49f8675\" (UID: \"3725842c-df7c-4b74-b6e5-6103c49f8675\") "
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.244488    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4c88fab-1746-46eb-b51a-1b8d1dd4e485" (UID: "f4c88fab-1746-46eb-b51a-1b8d1dd4e485"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.244590    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3725842c-df7c-4b74-b6e5-6103c49f8675-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3725842c-df7c-4b74-b6e5-6103c49f8675" (UID: "3725842c-df7c-4b74-b6e5-6103c49f8675"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.244207    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-metrics-certs\") pod \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") "
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.244676    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-openshift-service-ca\") pod \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") "
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.244756    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pp2w9\" (UniqueName: \"kubernetes.io/projected/3725842c-df7c-4b74-b6e5-6103c49f8675-kube-api-access-pp2w9\") pod \"3725842c-df7c-4b74-b6e5-6103c49f8675\" (UID: \"3725842c-df7c-4b74-b6e5-6103c49f8675\") "
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.244797    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2nx8\" (UniqueName: \"kubernetes.io/projected/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-kube-api-access-l2nx8\") pod \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\" (UID: \"27d6fbc2-2eb7-4403-973b-6b63ef7d188b\") "
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.244824    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhqzq\" (UniqueName: \"kubernetes.io/projected/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-kube-api-access-fhqzq\") pod \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\" (UID: \"f4c88fab-1746-46eb-b51a-1b8d1dd4e485\") "
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.245164    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.245183    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3725842c-df7c-4b74-b6e5-6103c49f8675-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.245231    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27d6fbc2-2eb7-4403-973b-6b63ef7d188b" (UID: "27d6fbc2-2eb7-4403-973b-6b63ef7d188b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.246513    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27d6fbc2-2eb7-4403-973b-6b63ef7d188b" (UID: "27d6fbc2-2eb7-4403-973b-6b63ef7d188b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:37:57.247553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.246982    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "27d6fbc2-2eb7-4403-973b-6b63ef7d188b" (UID: "27d6fbc2-2eb7-4403-973b-6b63ef7d188b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:37:57.248087 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.247675    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3725842c-df7c-4b74-b6e5-6103c49f8675-kube-api-access-pp2w9" (OuterVolumeSpecName: "kube-api-access-pp2w9") pod "3725842c-df7c-4b74-b6e5-6103c49f8675" (UID: "3725842c-df7c-4b74-b6e5-6103c49f8675"). InnerVolumeSpecName "kube-api-access-pp2w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:57.248087 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.247884    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-kube-api-access-fhqzq" (OuterVolumeSpecName: "kube-api-access-fhqzq") pod "f4c88fab-1746-46eb-b51a-1b8d1dd4e485" (UID: "f4c88fab-1746-46eb-b51a-1b8d1dd4e485"). InnerVolumeSpecName "kube-api-access-fhqzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:57.248589 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.248563    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-kube-api-access-l2nx8" (OuterVolumeSpecName: "kube-api-access-l2nx8") pod "27d6fbc2-2eb7-4403-973b-6b63ef7d188b" (UID: "27d6fbc2-2eb7-4403-973b-6b63ef7d188b"). InnerVolumeSpecName "kube-api-access-l2nx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:57.346282 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.346243    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp2w9\" (UniqueName: \"kubernetes.io/projected/3725842c-df7c-4b74-b6e5-6103c49f8675-kube-api-access-pp2w9\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:57.346282 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.346273    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2nx8\" (UniqueName: \"kubernetes.io/projected/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-kube-api-access-l2nx8\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:57.346282 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.346281    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqzq\" (UniqueName: \"kubernetes.io/projected/f4c88fab-1746-46eb-b51a-1b8d1dd4e485-kube-api-access-fhqzq\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:57.346282 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.346291    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:57.346624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.346299    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:57.346624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.346307    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d6fbc2-2eb7-4403-973b-6b63ef7d188b-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:37:57.658951 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.658916    2583 generic.go:356] "Generic (PLEG): container finished" podID="3725842c-df7c-4b74-b6e5-6103c49f8675" containerID="de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d" exitCode=0
Sep 06 10:37:57.659143 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.658959    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"
Sep 06 10:37:57.659143 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.659000    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227" event={"ID":"3725842c-df7c-4b74-b6e5-6103c49f8675","Type":"ContainerDied","Data":"de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d"}
Sep 06 10:37:57.659143 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.659063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227" event={"ID":"3725842c-df7c-4b74-b6e5-6103c49f8675","Type":"ContainerDied","Data":"ac7f7095e0555fdc19bc19d1d28399f4b9b646c0b634ad1c664b26a63871f8c4"}
Sep 06 10:37:57.659143 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.659097    2583 scope.go:117] "RemoveContainer" containerID="de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d"
Sep 06 10:37:57.660369 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.660318    2583 generic.go:356] "Generic (PLEG): container finished" podID="27d6fbc2-2eb7-4403-973b-6b63ef7d188b" containerID="bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6" exitCode=0
Sep 06 10:37:57.660502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.660369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm" event={"ID":"27d6fbc2-2eb7-4403-973b-6b63ef7d188b","Type":"ContainerDied","Data":"bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6"}
Sep 06 10:37:57.660502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.660429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm" event={"ID":"27d6fbc2-2eb7-4403-973b-6b63ef7d188b","Type":"ContainerDied","Data":"fce7f6dbf9d681e5e5ccfaf39bffa66de31e24622da4e04522139f645a5aa777"}
Sep 06 10:37:57.660502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.660381    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"
Sep 06 10:37:57.661833 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.661808    2583 generic.go:356] "Generic (PLEG): container finished" podID="f4c88fab-1746-46eb-b51a-1b8d1dd4e485" containerID="57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d" exitCode=0
Sep 06 10:37:57.661950 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.661851    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g" event={"ID":"f4c88fab-1746-46eb-b51a-1b8d1dd4e485","Type":"ContainerDied","Data":"57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d"}
Sep 06 10:37:57.661950 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.661907    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g" event={"ID":"f4c88fab-1746-46eb-b51a-1b8d1dd4e485","Type":"ContainerDied","Data":"3d3a2202d775be6d2dd7e76f3d0593c8c5195010d76e4f5c5bcc5e386f6a8c92"}
Sep 06 10:37:57.661950 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.661853    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"
Sep 06 10:37:57.670553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.669663    2583 scope.go:117] "RemoveContainer" containerID="de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d"
Sep 06 10:37:57.671655 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:37:57.670773    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d\": container with ID starting with de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d not found: ID does not exist" containerID="de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d"
Sep 06 10:37:57.671655 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.670808    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d"} err="failed to get container status \"de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d\": rpc error: code = NotFound desc = could not find container \"de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d\": container with ID starting with de6dc53dc56e36974f37ef24c40889794b403ef2dcbe1ef17e362f5e320f4f9d not found: ID does not exist"
Sep 06 10:37:57.671655 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.670831    2583 scope.go:117] "RemoveContainer" containerID="bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6"
Sep 06 10:37:57.680197 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.680178    2583 scope.go:117] "RemoveContainer" containerID="bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6"
Sep 06 10:37:57.680493 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:37:57.680473    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6\": container with ID starting with bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6 not found: ID does not exist" containerID="bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6"
Sep 06 10:37:57.680539 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.680501    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6"} err="failed to get container status \"bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6\": rpc error: code = NotFound desc = could not find container \"bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6\": container with ID starting with bf3c5dc2ee690f5d19348771624dbce7fda41719c40ca4fd818341aacca59ce6 not found: ID does not exist"
Sep 06 10:37:57.680539 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.680519    2583 scope.go:117] "RemoveContainer" containerID="57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d"
Sep 06 10:37:57.688044 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.687976    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"]
Sep 06 10:37:57.688781 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.688761    2583 scope.go:117] "RemoveContainer" containerID="57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d"
Sep 06 10:37:57.689086 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:37:57.689066    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d\": container with ID starting with 57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d not found: ID does not exist" containerID="57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d"
Sep 06 10:37:57.689162 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.689093    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d"} err="failed to get container status \"57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d\": rpc error: code = NotFound desc = could not find container \"57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d\": container with ID starting with 57bcd1f65d27b6c4bfea79f5f2a1db57094e93b9db96c3ad78d8bd67f201f35d not found: ID does not exist"
Sep 06 10:37:57.690061 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.690037    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-04526a-67d748f585-mb2rm"]
Sep 06 10:37:57.698953 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.698925    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"]
Sep 06 10:37:57.703311 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.703288    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-04526a-596594d78f-jp227"]
Sep 06 10:37:57.720585 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.720555    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"]
Sep 06 10:37:57.724712 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:57.724683    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g"]
Sep 06 10:37:58.169970 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:58.169928    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-04526a-6b9dbc9f8b-w2b7g" podUID="f4c88fab-1746-46eb-b51a-1b8d1dd4e485" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.37:9004: i/o timeout"
Sep 06 10:37:59.247299 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:59.247268    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d6fbc2-2eb7-4403-973b-6b63ef7d188b" path="/var/lib/kubelet/pods/27d6fbc2-2eb7-4403-973b-6b63ef7d188b/volumes"
Sep 06 10:37:59.247716 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:59.247690    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3725842c-df7c-4b74-b6e5-6103c49f8675" path="/var/lib/kubelet/pods/3725842c-df7c-4b74-b6e5-6103c49f8675/volumes"
Sep 06 10:37:59.248099 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:37:59.248081    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4c88fab-1746-46eb-b51a-1b8d1dd4e485" path="/var/lib/kubelet/pods/f4c88fab-1746-46eb-b51a-1b8d1dd4e485/volumes"
Sep 06 10:38:04.374848 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:04.374813    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:38:14.557163 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:14.557075    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"]
Sep 06 10:38:14.557973 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:14.557934    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds" podUID="7b75eb94-51f1-4af4-9e26-42ec34d91b6b" containerName="payload-pre-processing" containerID="cri-o://93340deb0476c15b43ad7735490410fa3e48594bd8dc26002a04ad2997ee815a" gracePeriod=30
Sep 06 10:38:14.744449 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:14.744417    2583 generic.go:356] "Generic (PLEG): container finished" podID="7b75eb94-51f1-4af4-9e26-42ec34d91b6b" containerID="93340deb0476c15b43ad7735490410fa3e48594bd8dc26002a04ad2997ee815a" exitCode=0
Sep 06 10:38:14.744613 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:14.744491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds" event={"ID":"7b75eb94-51f1-4af4-9e26-42ec34d91b6b","Type":"ContainerDied","Data":"93340deb0476c15b43ad7735490410fa3e48594bd8dc26002a04ad2997ee815a"}
Sep 06 10:38:14.810246 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:14.810181    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:38:14.904173 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:14.904135    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqlk2\" (UniqueName: \"kubernetes.io/projected/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-kube-api-access-rqlk2\") pod \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\" (UID: \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\") "
Sep 06 10:38:14.904404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:14.904259    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-plugins-config-volume\") pod \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\" (UID: \"7b75eb94-51f1-4af4-9e26-42ec34d91b6b\") "
Sep 06 10:38:14.904654 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:14.904632    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7b75eb94-51f1-4af4-9e26-42ec34d91b6b" (UID: "7b75eb94-51f1-4af4-9e26-42ec34d91b6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:38:14.906285 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:14.906255    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-kube-api-access-rqlk2" (OuterVolumeSpecName: "kube-api-access-rqlk2") pod "7b75eb94-51f1-4af4-9e26-42ec34d91b6b" (UID: "7b75eb94-51f1-4af4-9e26-42ec34d91b6b"). InnerVolumeSpecName "kube-api-access-rqlk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:38:15.005877 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:15.005833    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:38:15.005877 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:15.005867    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqlk2\" (UniqueName: \"kubernetes.io/projected/7b75eb94-51f1-4af4-9e26-42ec34d91b6b-kube-api-access-rqlk2\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:38:15.750642 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:15.750613    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"
Sep 06 10:38:15.751099 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:15.750641    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds" event={"ID":"7b75eb94-51f1-4af4-9e26-42ec34d91b6b","Type":"ContainerDied","Data":"20025a9f03911ae5f63ce7a0d24d8202cfa8dad424029f118bca01284006fc1a"}
Sep 06 10:38:15.751099 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:15.750694    2583 scope.go:117] "RemoveContainer" containerID="93340deb0476c15b43ad7735490410fa3e48594bd8dc26002a04ad2997ee815a"
Sep 06 10:38:15.766073 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:15.766046    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"]
Sep 06 10:38:15.772168 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:15.772139    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bd3407-57c64c58cd-crbds"]
Sep 06 10:38:17.239497 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:17.239457    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b75eb94-51f1-4af4-9e26-42ec34d91b6b" path="/var/lib/kubelet/pods/7b75eb94-51f1-4af4-9e26-42ec34d91b6b/volumes"
Sep 06 10:38:18.071744 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:18.071710    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:38:19.285009 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.284977    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"]
Sep 06 10:38:19.285507 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285488    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b75eb94-51f1-4af4-9e26-42ec34d91b6b" containerName="payload-pre-processing"
Sep 06 10:38:19.285627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285510    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b75eb94-51f1-4af4-9e26-42ec34d91b6b" containerName="payload-pre-processing"
Sep 06 10:38:19.285627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285529    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67da3e2d-13d8-4aa1-8b27-f444145d31ae" containerName="maas-api"
Sep 06 10:38:19.285627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285538    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="67da3e2d-13d8-4aa1-8b27-f444145d31ae" containerName="maas-api"
Sep 06 10:38:19.285627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285557    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3725842c-df7c-4b74-b6e5-6103c49f8675" containerName="payload-processing"
Sep 06 10:38:19.285627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285566    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3725842c-df7c-4b74-b6e5-6103c49f8675" containerName="payload-processing"
Sep 06 10:38:19.285627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285579    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4c88fab-1746-46eb-b51a-1b8d1dd4e485" containerName="payload-pre-processing"
Sep 06 10:38:19.285627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285588    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4c88fab-1746-46eb-b51a-1b8d1dd4e485" containerName="payload-pre-processing"
Sep 06 10:38:19.285627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285603    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d6fbc2-2eb7-4403-973b-6b63ef7d188b" containerName="maas-api"
Sep 06 10:38:19.285627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285612    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d6fbc2-2eb7-4403-973b-6b63ef7d188b" containerName="maas-api"
Sep 06 10:38:19.286040 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285701    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d6fbc2-2eb7-4403-973b-6b63ef7d188b" containerName="maas-api"
Sep 06 10:38:19.286040 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285715    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b75eb94-51f1-4af4-9e26-42ec34d91b6b" containerName="payload-pre-processing"
Sep 06 10:38:19.286040 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285724    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4c88fab-1746-46eb-b51a-1b8d1dd4e485" containerName="payload-pre-processing"
Sep 06 10:38:19.286040 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285734    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="67da3e2d-13d8-4aa1-8b27-f444145d31ae" containerName="maas-api"
Sep 06 10:38:19.286040 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.285743    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3725842c-df7c-4b74-b6e5-6103c49f8675" containerName="payload-processing"
Sep 06 10:38:19.289122 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.289103    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.291249 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.291221    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-27a2b9-serving-cert\""
Sep 06 10:38:19.299649 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.299623    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"]
Sep 06 10:38:19.348288 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.348255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwmr\" (UniqueName: \"kubernetes.io/projected/218e9039-ceee-4386-aed4-96a1e3f71d2b-kube-api-access-wkwmr\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.348524 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.348391    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.348524 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.348445    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.348524 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.348475    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218e9039-ceee-4386-aed4-96a1e3f71d2b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.448922 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.448882    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.448922 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.448926    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218e9039-ceee-4386-aed4-96a1e3f71d2b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.449133 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.448959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwmr\" (UniqueName: \"kubernetes.io/projected/218e9039-ceee-4386-aed4-96a1e3f71d2b-kube-api-access-wkwmr\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.449133 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.449004    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.449133 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:38:19.449101    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-serving-cert: secret "maas-api-e2e-shared-a-w2-27a2b9-serving-cert" not found
Sep 06 10:38:19.449254 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:38:19.449158    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-maas-api-tls podName:218e9039-ceee-4386-aed4-96a1e3f71d2b nodeName:}" failed. No retries permitted until 2026-09-06 10:38:19.94914439 +0000 UTC m=+991.221785871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8" (UID: "218e9039-ceee-4386-aed4-96a1e3f71d2b") : secret "maas-api-e2e-shared-a-w2-27a2b9-serving-cert" not found
Sep 06 10:38:19.449706 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.449683    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218e9039-ceee-4386-aed4-96a1e3f71d2b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.451351 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.451317    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.459087 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.459061    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwmr\" (UniqueName: \"kubernetes.io/projected/218e9039-ceee-4386-aed4-96a1e3f71d2b-kube-api-access-wkwmr\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.946194 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.946154    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"]
Sep 06 10:38:19.949576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.949554    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:19.951519 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.951484    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-27a2b9\""
Sep 06 10:38:19.951519 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.951508    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-27a2b9-dockercfg-cgwjz\""
Sep 06 10:38:19.952707 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.952686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.955292 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.955268    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:19.960106 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:19.960074    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"]
Sep 06 10:38:20.054104 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.054068    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnb4\" (UniqueName: \"kubernetes.io/projected/4837282d-e2ac-40b2-a8cf-d311cbe18584-kube-api-access-2bnb4\") pod \"payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc\" (UID: \"4837282d-e2ac-40b2-a8cf-d311cbe18584\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:20.054320 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.054238    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4837282d-e2ac-40b2-a8cf-d311cbe18584-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc\" (UID: \"4837282d-e2ac-40b2-a8cf-d311cbe18584\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:20.155396 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.155357    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4837282d-e2ac-40b2-a8cf-d311cbe18584-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc\" (UID: \"4837282d-e2ac-40b2-a8cf-d311cbe18584\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:20.155645 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.155410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnb4\" (UniqueName: \"kubernetes.io/projected/4837282d-e2ac-40b2-a8cf-d311cbe18584-kube-api-access-2bnb4\") pod \"payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc\" (UID: \"4837282d-e2ac-40b2-a8cf-d311cbe18584\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:20.156091 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.156068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4837282d-e2ac-40b2-a8cf-d311cbe18584-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc\" (UID: \"4837282d-e2ac-40b2-a8cf-d311cbe18584\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:20.162834 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.162803    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnb4\" (UniqueName: \"kubernetes.io/projected/4837282d-e2ac-40b2-a8cf-d311cbe18584-kube-api-access-2bnb4\") pod \"payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc\" (UID: \"4837282d-e2ac-40b2-a8cf-d311cbe18584\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:20.200653 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.200572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:20.268784 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.268752    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:20.472311 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.472284    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"]
Sep 06 10:38:20.474814 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:38:20.474778    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218e9039_ceee_4386_aed4_96a1e3f71d2b.slice/crio-7c738c14a2390a40e0e82051b86fc62c6dd83e276c15ca889e06bf507867ab2d WatchSource:0}: Error finding container 7c738c14a2390a40e0e82051b86fc62c6dd83e276c15ca889e06bf507867ab2d: Status 404 returned error can't find the container with id 7c738c14a2390a40e0e82051b86fc62c6dd83e276c15ca889e06bf507867ab2d
Sep 06 10:38:20.476156 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.476136    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:38:20.541139 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.541112    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"]
Sep 06 10:38:20.543546 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:38:20.543507    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4837282d_e2ac_40b2_a8cf_d311cbe18584.slice/crio-4bdd0807227bdeb7c02b9bc810dc0604daf603f657978b53a0db612cfa8889a9 WatchSource:0}: Error finding container 4bdd0807227bdeb7c02b9bc810dc0604daf603f657978b53a0db612cfa8889a9: Status 404 returned error can't find the container with id 4bdd0807227bdeb7c02b9bc810dc0604daf603f657978b53a0db612cfa8889a9
Sep 06 10:38:20.776494 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.776394    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc" event={"ID":"4837282d-e2ac-40b2-a8cf-d311cbe18584","Type":"ContainerStarted","Data":"fa3640b307663c74dfa39ca674e7796eb5d01c0939edba1b2fce745bb4710c1a"}
Sep 06 10:38:20.776494 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.776455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc" event={"ID":"4837282d-e2ac-40b2-a8cf-d311cbe18584","Type":"ContainerStarted","Data":"4bdd0807227bdeb7c02b9bc810dc0604daf603f657978b53a0db612cfa8889a9"}
Sep 06 10:38:20.776720 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.776506    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:20.777678 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.777654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8" event={"ID":"218e9039-ceee-4386-aed4-96a1e3f71d2b","Type":"ContainerStarted","Data":"7c738c14a2390a40e0e82051b86fc62c6dd83e276c15ca889e06bf507867ab2d"}
Sep 06 10:38:20.794661 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:20.794603    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc" podStartSLOduration=1.794583426 podStartE2EDuration="1.794583426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:20.7919266 +0000 UTC m=+992.064568102" watchObservedRunningTime="2026-09-06 10:38:20.794583426 +0000 UTC m=+992.067224929"
Sep 06 10:38:21.784507 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:21.784452    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8" event={"ID":"218e9039-ceee-4386-aed4-96a1e3f71d2b","Type":"ContainerStarted","Data":"fb3f7bbe2bed655864652ecbca9f87e6c9c0031c3870019bebe34cf3b839a0ae"}
Sep 06 10:38:21.800658 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:21.800595    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8" podStartSLOduration=2.429355649 podStartE2EDuration="2.800575635s" totalImagesPullingTime="371.219986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:21.800241394 +0000 UTC m=+993.072882898" watchObservedRunningTime="2026-09-06 10:38:21.800575635 +0000 UTC m=+993.073217142"
Sep 06 10:38:22.789547 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:22.789516    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:27.696292 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.696254    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"]
Sep 06 10:38:27.699791 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.699770    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:27.702014 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.701992    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-6751d7\""
Sep 06 10:38:27.712165 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.712138    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"]
Sep 06 10:38:27.825223 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.825116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5405b19e-e7df-4afd-8cfe-f4a48479f728-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h\" (UID: \"5405b19e-e7df-4afd-8cfe-f4a48479f728\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:27.825398 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.825255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2bb9\" (UniqueName: \"kubernetes.io/projected/5405b19e-e7df-4afd-8cfe-f4a48479f728-kube-api-access-j2bb9\") pod \"payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h\" (UID: \"5405b19e-e7df-4afd-8cfe-f4a48479f728\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:27.926246 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.926198    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5405b19e-e7df-4afd-8cfe-f4a48479f728-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h\" (UID: \"5405b19e-e7df-4afd-8cfe-f4a48479f728\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:27.926479 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.926319    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2bb9\" (UniqueName: \"kubernetes.io/projected/5405b19e-e7df-4afd-8cfe-f4a48479f728-kube-api-access-j2bb9\") pod \"payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h\" (UID: \"5405b19e-e7df-4afd-8cfe-f4a48479f728\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:27.927094 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.927068    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5405b19e-e7df-4afd-8cfe-f4a48479f728-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h\" (UID: \"5405b19e-e7df-4afd-8cfe-f4a48479f728\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:27.934315 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:27.934280    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2bb9\" (UniqueName: \"kubernetes.io/projected/5405b19e-e7df-4afd-8cfe-f4a48479f728-kube-api-access-j2bb9\") pod \"payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h\" (UID: \"5405b19e-e7df-4afd-8cfe-f4a48479f728\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:28.011099 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:28.011061    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:28.282116 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:28.281986    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"]
Sep 06 10:38:28.284914 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:38:28.284888    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5405b19e_e7df_4afd_8cfe_f4a48479f728.slice/crio-915f03a9986febc51f4ad6b80cc9cb9eae4b90d5a223d7318d42cb9c2d873538 WatchSource:0}: Error finding container 915f03a9986febc51f4ad6b80cc9cb9eae4b90d5a223d7318d42cb9c2d873538: Status 404 returned error can't find the container with id 915f03a9986febc51f4ad6b80cc9cb9eae4b90d5a223d7318d42cb9c2d873538
Sep 06 10:38:28.798962 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:28.798935    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:38:28.829519 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:28.829477    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h" event={"ID":"5405b19e-e7df-4afd-8cfe-f4a48479f728","Type":"ContainerStarted","Data":"1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd"}
Sep 06 10:38:28.829707 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:28.829601    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h" event={"ID":"5405b19e-e7df-4afd-8cfe-f4a48479f728","Type":"ContainerStarted","Data":"915f03a9986febc51f4ad6b80cc9cb9eae4b90d5a223d7318d42cb9c2d873538"}
Sep 06 10:38:28.829797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:28.829765    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:28.847432 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:28.847328    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h" podStartSLOduration=1.8473069949999998 podStartE2EDuration="1.847306995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:28.84688466 +0000 UTC m=+1000.119526162" watchObservedRunningTime="2026-09-06 10:38:28.847306995 +0000 UTC m=+1000.119948498"
Sep 06 10:38:30.479084 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:30.479042    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:38:31.787504 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:31.787467    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:38:39.836946 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:39.836914    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:38:49.288295 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.288263    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"]
Sep 06 10:38:49.291594 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.291572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:38:49.293546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.293524    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-71e210-dockercfg-wdc2l\""
Sep 06 10:38:49.293669 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.293524    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-71e210\""
Sep 06 10:38:49.299892 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.299850    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"]
Sep 06 10:38:49.306307 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.306283    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"]
Sep 06 10:38:49.311015 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.310993    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:38:49.320379 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.320318    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"]
Sep 06 10:38:49.423591 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.423555    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcv4\" (UniqueName: \"kubernetes.io/projected/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-kube-api-access-gtcv4\") pod \"payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t\" (UID: \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:38:49.423591 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.423603    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t\" (UID: \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:38:49.423803 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.423635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zh4v\" (UniqueName: \"kubernetes.io/projected/8e5bef5e-4131-452a-bb36-0bb0246a703d-kube-api-access-8zh4v\") pod \"payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw\" (UID: \"8e5bef5e-4131-452a-bb36-0bb0246a703d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:38:49.423803 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.423683    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5bef5e-4131-452a-bb36-0bb0246a703d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw\" (UID: \"8e5bef5e-4131-452a-bb36-0bb0246a703d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:38:49.524290 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.524255    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcv4\" (UniqueName: \"kubernetes.io/projected/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-kube-api-access-gtcv4\") pod \"payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t\" (UID: \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:38:49.524528 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.524306    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t\" (UID: \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:38:49.524528 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.524367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zh4v\" (UniqueName: \"kubernetes.io/projected/8e5bef5e-4131-452a-bb36-0bb0246a703d-kube-api-access-8zh4v\") pod \"payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw\" (UID: \"8e5bef5e-4131-452a-bb36-0bb0246a703d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:38:49.524528 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.524399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5bef5e-4131-452a-bb36-0bb0246a703d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw\" (UID: \"8e5bef5e-4131-452a-bb36-0bb0246a703d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:38:49.525373 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.525328    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5bef5e-4131-452a-bb36-0bb0246a703d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw\" (UID: \"8e5bef5e-4131-452a-bb36-0bb0246a703d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:38:49.525373 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.525328    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t\" (UID: \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:38:49.532305 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.532275    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcv4\" (UniqueName: \"kubernetes.io/projected/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-kube-api-access-gtcv4\") pod \"payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t\" (UID: \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:38:49.532305 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.532293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zh4v\" (UniqueName: \"kubernetes.io/projected/8e5bef5e-4131-452a-bb36-0bb0246a703d-kube-api-access-8zh4v\") pod \"payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw\" (UID: \"8e5bef5e-4131-452a-bb36-0bb0246a703d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:38:49.604528 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.604489    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:38:49.625723 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.625690    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:38:49.883066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.882956    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"]
Sep 06 10:38:49.885716 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:38:49.885683    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cae6352_40a3_4c6b_a0af_b43d01ec3e42.slice/crio-2aa64cdf9aa857d42dce3b16188323ceb0f87397f287e1bbf91d440210f6624e WatchSource:0}: Error finding container 2aa64cdf9aa857d42dce3b16188323ceb0f87397f287e1bbf91d440210f6624e: Status 404 returned error can't find the container with id 2aa64cdf9aa857d42dce3b16188323ceb0f87397f287e1bbf91d440210f6624e
Sep 06 10:38:49.914661 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.914634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"]
Sep 06 10:38:49.917445 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:38:49.917416    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5bef5e_4131_452a_bb36_0bb0246a703d.slice/crio-48ae3542a1cf37fa29e40e876d6ccbee7ac0f81e7d186236c769b180f2edea6f WatchSource:0}: Error finding container 48ae3542a1cf37fa29e40e876d6ccbee7ac0f81e7d186236c769b180f2edea6f: Status 404 returned error can't find the container with id 48ae3542a1cf37fa29e40e876d6ccbee7ac0f81e7d186236c769b180f2edea6f
Sep 06 10:38:49.936649 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.936619    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw" event={"ID":"8e5bef5e-4131-452a-bb36-0bb0246a703d","Type":"ContainerStarted","Data":"48ae3542a1cf37fa29e40e876d6ccbee7ac0f81e7d186236c769b180f2edea6f"}
Sep 06 10:38:49.938081 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:49.938050    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t" event={"ID":"5cae6352-40a3-4c6b-a0af-b43d01ec3e42","Type":"ContainerStarted","Data":"2aa64cdf9aa857d42dce3b16188323ceb0f87397f287e1bbf91d440210f6624e"}
Sep 06 10:38:50.944702 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:50.944661    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw" event={"ID":"8e5bef5e-4131-452a-bb36-0bb0246a703d","Type":"ContainerStarted","Data":"817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175"}
Sep 06 10:38:50.945178 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:50.944784    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:38:50.946454 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:50.946428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t" event={"ID":"5cae6352-40a3-4c6b-a0af-b43d01ec3e42","Type":"ContainerStarted","Data":"a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9"}
Sep 06 10:38:50.946545 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:50.946473    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:38:50.960089 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:50.960031    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw" podStartSLOduration=1.9599994729999999 podStartE2EDuration="1.959999473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:50.959818864 +0000 UTC m=+1022.232460367" watchObservedRunningTime="2026-09-06 10:38:50.959999473 +0000 UTC m=+1022.232640978"
Sep 06 10:38:50.975124 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:50.975073    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t" podStartSLOduration=1.975055928 podStartE2EDuration="1.975055928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:50.973423472 +0000 UTC m=+1022.246064988" watchObservedRunningTime="2026-09-06 10:38:50.975055928 +0000 UTC m=+1022.247697481"
Sep 06 10:38:51.680238 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:51.680197    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:38:57.005098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.005057    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"]
Sep 06 10:38:57.009637 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.009613    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:38:57.012254 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.012228    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-682ea0\""
Sep 06 10:38:57.021574 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.021546    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"]
Sep 06 10:38:57.044808 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.044777    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"]
Sep 06 10:38:57.048291 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.048266    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:38:57.050212 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.050192    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-682ea0-dockercfg-ph8mz\""
Sep 06 10:38:57.057074 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.057047    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"]
Sep 06 10:38:57.094445 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.094404    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2658480c-969b-42c4-b007-3ea9794e1dd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl\" (UID: \"2658480c-969b-42c4-b007-3ea9794e1dd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:38:57.094624 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.094514    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvw9d\" (UniqueName: \"kubernetes.io/projected/2658480c-969b-42c4-b007-3ea9794e1dd3-kube-api-access-kvw9d\") pod \"payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl\" (UID: \"2658480c-969b-42c4-b007-3ea9794e1dd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:38:57.195681 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.195646    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2658480c-969b-42c4-b007-3ea9794e1dd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl\" (UID: \"2658480c-969b-42c4-b007-3ea9794e1dd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:38:57.195852 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.195699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvw9d\" (UniqueName: \"kubernetes.io/projected/2658480c-969b-42c4-b007-3ea9794e1dd3-kube-api-access-kvw9d\") pod \"payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl\" (UID: \"2658480c-969b-42c4-b007-3ea9794e1dd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:38:57.195852 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.195739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g\" (UID: \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:38:57.195852 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.195769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwfv\" (UniqueName: \"kubernetes.io/projected/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-kube-api-access-gnwfv\") pod \"payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g\" (UID: \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:38:57.196360 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.196316    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2658480c-969b-42c4-b007-3ea9794e1dd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl\" (UID: \"2658480c-969b-42c4-b007-3ea9794e1dd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:38:57.203299 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.203272    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvw9d\" (UniqueName: \"kubernetes.io/projected/2658480c-969b-42c4-b007-3ea9794e1dd3-kube-api-access-kvw9d\") pod \"payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl\" (UID: \"2658480c-969b-42c4-b007-3ea9794e1dd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:38:57.296573 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.296478    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g\" (UID: \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:38:57.296573 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.296527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwfv\" (UniqueName: \"kubernetes.io/projected/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-kube-api-access-gnwfv\") pod \"payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g\" (UID: \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:38:57.297251 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.297229    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g\" (UID: \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:38:57.303979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.303955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwfv\" (UniqueName: \"kubernetes.io/projected/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-kube-api-access-gnwfv\") pod \"payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g\" (UID: \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:38:57.321803 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.321762    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:38:57.359263 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.359225    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:38:57.597849 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.597820    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"]
Sep 06 10:38:57.600545 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:38:57.600517    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2658480c_969b_42c4_b007_3ea9794e1dd3.slice/crio-9fdf1234d2d4fa773d6f5be14baf1d96293fd726704daef694ac5c08954cbde2 WatchSource:0}: Error finding container 9fdf1234d2d4fa773d6f5be14baf1d96293fd726704daef694ac5c08954cbde2: Status 404 returned error can't find the container with id 9fdf1234d2d4fa773d6f5be14baf1d96293fd726704daef694ac5c08954cbde2
Sep 06 10:38:57.643700 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.643677    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"]
Sep 06 10:38:57.646448 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:38:57.646419    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb5e456_c0c5_42b6_bafa_ad4662cce6bf.slice/crio-0e042a6a2538c50e280128b30c8d6afdcfb57f70187452bc9d4507d207eb76a6 WatchSource:0}: Error finding container 0e042a6a2538c50e280128b30c8d6afdcfb57f70187452bc9d4507d207eb76a6: Status 404 returned error can't find the container with id 0e042a6a2538c50e280128b30c8d6afdcfb57f70187452bc9d4507d207eb76a6
Sep 06 10:38:57.982090 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.981987    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g" event={"ID":"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf","Type":"ContainerStarted","Data":"cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a"}
Sep 06 10:38:57.982090 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.982050    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g" event={"ID":"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf","Type":"ContainerStarted","Data":"0e042a6a2538c50e280128b30c8d6afdcfb57f70187452bc9d4507d207eb76a6"}
Sep 06 10:38:57.982409 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.982195    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:38:57.983983 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.983955    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl" event={"ID":"2658480c-969b-42c4-b007-3ea9794e1dd3","Type":"ContainerStarted","Data":"355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037"}
Sep 06 10:38:57.984108 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.984002    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl" event={"ID":"2658480c-969b-42c4-b007-3ea9794e1dd3","Type":"ContainerStarted","Data":"9fdf1234d2d4fa773d6f5be14baf1d96293fd726704daef694ac5c08954cbde2"}
Sep 06 10:38:57.984108 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.984087    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:38:57.998905 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:57.998853    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g" podStartSLOduration=0.998835368 podStartE2EDuration="998.835368ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:57.997591463 +0000 UTC m=+1029.270232966" watchObservedRunningTime="2026-09-06 10:38:57.998835368 +0000 UTC m=+1029.271476871"
Sep 06 10:38:58.013593 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:38:58.013540    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl" podStartSLOduration=2.013523463 podStartE2EDuration="2.013523463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:58.011546002 +0000 UTC m=+1029.284187509" watchObservedRunningTime="2026-09-06 10:38:58.013523463 +0000 UTC m=+1029.286164966"
Sep 06 10:39:01.954818 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:01.954783    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:39:01.955259 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:01.954857    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:39:05.782144 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:05.782088    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:39:08.991197 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:08.991166    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:39:08.991700 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:08.991248    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:39:32.086927 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.086889    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"]
Sep 06 10:39:32.091623 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.091601    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:32.095661 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.095638    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b21e0db6\""
Sep 06 10:39:32.095790 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.095729    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b21e0db6-dockercfg-cv8ss\""
Sep 06 10:39:32.105475 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.105449    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"]
Sep 06 10:39:32.120045 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.120009    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ac05bd-e877-4919-9331-f88ecaad9875-plugins-config-volume\") pod \"payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws\" (UID: \"16ac05bd-e877-4919-9331-f88ecaad9875\") " pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:32.120189 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.120116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2vw\" (UniqueName: \"kubernetes.io/projected/16ac05bd-e877-4919-9331-f88ecaad9875-kube-api-access-cv2vw\") pod \"payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws\" (UID: \"16ac05bd-e877-4919-9331-f88ecaad9875\") " pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:32.221637 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.221599    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2vw\" (UniqueName: \"kubernetes.io/projected/16ac05bd-e877-4919-9331-f88ecaad9875-kube-api-access-cv2vw\") pod \"payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws\" (UID: \"16ac05bd-e877-4919-9331-f88ecaad9875\") " pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:32.221851 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.221704    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ac05bd-e877-4919-9331-f88ecaad9875-plugins-config-volume\") pod \"payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws\" (UID: \"16ac05bd-e877-4919-9331-f88ecaad9875\") " pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:32.222763 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.222735    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ac05bd-e877-4919-9331-f88ecaad9875-plugins-config-volume\") pod \"payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws\" (UID: \"16ac05bd-e877-4919-9331-f88ecaad9875\") " pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:32.229588 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.229557    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2vw\" (UniqueName: \"kubernetes.io/projected/16ac05bd-e877-4919-9331-f88ecaad9875-kube-api-access-cv2vw\") pod \"payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws\" (UID: \"16ac05bd-e877-4919-9331-f88ecaad9875\") " pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:32.402604 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.402514    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:32.676491 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:32.676463    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"]
Sep 06 10:39:32.678751 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:39:32.678723    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ac05bd_e877_4919_9331_f88ecaad9875.slice/crio-c88e13cc89ff54a4787f7d40f29f39eb2564683fd4ce82a18b1c15a785b7dc49 WatchSource:0}: Error finding container c88e13cc89ff54a4787f7d40f29f39eb2564683fd4ce82a18b1c15a785b7dc49: Status 404 returned error can't find the container with id c88e13cc89ff54a4787f7d40f29f39eb2564683fd4ce82a18b1c15a785b7dc49
Sep 06 10:39:33.157727 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:33.157692    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws" event={"ID":"16ac05bd-e877-4919-9331-f88ecaad9875","Type":"ContainerStarted","Data":"54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551"}
Sep 06 10:39:33.158218 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:33.157748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws" event={"ID":"16ac05bd-e877-4919-9331-f88ecaad9875","Type":"ContainerStarted","Data":"c88e13cc89ff54a4787f7d40f29f39eb2564683fd4ce82a18b1c15a785b7dc49"}
Sep 06 10:39:33.158218 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:33.157777    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:33.172382 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:33.172279    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws" podStartSLOduration=1.172258454 podStartE2EDuration="1.172258454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:39:33.171906018 +0000 UTC m=+1064.444547521" watchObservedRunningTime="2026-09-06 10:39:33.172258454 +0000 UTC m=+1064.444899956"
Sep 06 10:39:38.173136 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:38.173094    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:39:44.164575 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:44.164488    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:39:49.538697 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:39:49.538654    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:40:12.588031 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:12.587990    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:40:17.300256 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.300218    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"]
Sep 06 10:40:17.304280 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.304260    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.307040 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.307010    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"default-dockercfg-p7bg2\""
Sep 06 10:40:17.307172 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.307014    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"auth-test-model-682ea0-kserve-self-signed-certs\""
Sep 06 10:40:17.307172 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.307014    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"openshift-service-ca.crt\""
Sep 06 10:40:17.307172 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.307103    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"kube-root-ca.crt\""
Sep 06 10:40:17.312856 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.312468    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"]
Sep 06 10:40:17.321579 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.321549    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-home\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.321822 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.321799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-tmp-dir\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.321976 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.321958    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a356caa-cf45-4b8f-a74e-d489750dda80-tls-certs\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.322098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.322081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7qln\" (UniqueName: \"kubernetes.io/projected/6a356caa-cf45-4b8f-a74e-d489750dda80-kube-api-access-v7qln\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.322255 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.322240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-dshm\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.322437 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.322412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-model-cache\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.423879 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.423835    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-tmp-dir\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.424075 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.423889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a356caa-cf45-4b8f-a74e-d489750dda80-tls-certs\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.424075 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.423922    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7qln\" (UniqueName: \"kubernetes.io/projected/6a356caa-cf45-4b8f-a74e-d489750dda80-kube-api-access-v7qln\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.424210 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.424093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-dshm\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.424210 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.424133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-model-cache\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.424316 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.424215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-home\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.424316 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.424297    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-tmp-dir\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.424563 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.424542    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-model-cache\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.424563 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.424555    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-home\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.426348 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.426311    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-dshm\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.426610 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.426590    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a356caa-cf45-4b8f-a74e-d489750dda80-tls-certs\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.432122 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.432092    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7qln\" (UniqueName: \"kubernetes.io/projected/6a356caa-cf45-4b8f-a74e-d489750dda80-kube-api-access-v7qln\") pod \"auth-test-model-682ea0-kserve-dcb88cd67-f24s6\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.616434 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.616396    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:17.892013 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:17.891987    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"]
Sep 06 10:40:18.384914 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:18.384879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6" event={"ID":"6a356caa-cf45-4b8f-a74e-d489750dda80","Type":"ContainerStarted","Data":"b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5"}
Sep 06 10:40:18.384914 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:18.384922    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6" event={"ID":"6a356caa-cf45-4b8f-a74e-d489750dda80","Type":"ContainerStarted","Data":"691c9888cfcb1bd74f87a6d23ce0850a33aa613b2e7bd92d8eb4fea88f495fe9"}
Sep 06 10:40:18.401154 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:18.401096    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6" podStartSLOduration=1.4010802660000001 podStartE2EDuration="1.401080266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:40:18.399973833 +0000 UTC m=+1109.672615358" watchObservedRunningTime="2026-09-06 10:40:18.401080266 +0000 UTC m=+1109.673721768"
Sep 06 10:40:27.616829 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:27.616792    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:27.617276 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:27.617056    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:27.635787 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:27.635760    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:28.441717 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:28.441689    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:40:32.277067 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:32.277029    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:40:45.972542 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:40:45.972493    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:41:04.876522 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:04.876482    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:41:05.698287 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:05.698248    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"]
Sep 06 10:41:05.698539 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:05.698508    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws" podUID="16ac05bd-e877-4919-9331-f88ecaad9875" containerName="payload-processing" containerID="cri-o://54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551" gracePeriod=30
Sep 06 10:41:05.969475 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:05.969451    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:41:06.066139 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.066110    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ac05bd-e877-4919-9331-f88ecaad9875-plugins-config-volume\") pod \"16ac05bd-e877-4919-9331-f88ecaad9875\" (UID: \"16ac05bd-e877-4919-9331-f88ecaad9875\") "
Sep 06 10:41:06.066365 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.066153    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv2vw\" (UniqueName: \"kubernetes.io/projected/16ac05bd-e877-4919-9331-f88ecaad9875-kube-api-access-cv2vw\") pod \"16ac05bd-e877-4919-9331-f88ecaad9875\" (UID: \"16ac05bd-e877-4919-9331-f88ecaad9875\") "
Sep 06 10:41:06.066517 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.066490    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ac05bd-e877-4919-9331-f88ecaad9875-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16ac05bd-e877-4919-9331-f88ecaad9875" (UID: "16ac05bd-e877-4919-9331-f88ecaad9875"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:41:06.068279 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.068252    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ac05bd-e877-4919-9331-f88ecaad9875-kube-api-access-cv2vw" (OuterVolumeSpecName: "kube-api-access-cv2vw") pod "16ac05bd-e877-4919-9331-f88ecaad9875" (UID: "16ac05bd-e877-4919-9331-f88ecaad9875"). InnerVolumeSpecName "kube-api-access-cv2vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:41:06.167734 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.167693    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ac05bd-e877-4919-9331-f88ecaad9875-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:41:06.167734 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.167727    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2vw\" (UniqueName: \"kubernetes.io/projected/16ac05bd-e877-4919-9331-f88ecaad9875-kube-api-access-cv2vw\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:41:06.615585 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.615548    2583 generic.go:356] "Generic (PLEG): container finished" podID="16ac05bd-e877-4919-9331-f88ecaad9875" containerID="54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551" exitCode=0
Sep 06 10:41:06.615774 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.615591    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"
Sep 06 10:41:06.615774 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.615634    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws" event={"ID":"16ac05bd-e877-4919-9331-f88ecaad9875","Type":"ContainerDied","Data":"54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551"}
Sep 06 10:41:06.615774 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.615688    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws" event={"ID":"16ac05bd-e877-4919-9331-f88ecaad9875","Type":"ContainerDied","Data":"c88e13cc89ff54a4787f7d40f29f39eb2564683fd4ce82a18b1c15a785b7dc49"}
Sep 06 10:41:06.615774 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.615712    2583 scope.go:117] "RemoveContainer" containerID="54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551"
Sep 06 10:41:06.625266 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.625230    2583 scope.go:117] "RemoveContainer" containerID="54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551"
Sep 06 10:41:06.625629 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:41:06.625609    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551\": container with ID starting with 54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551 not found: ID does not exist" containerID="54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551"
Sep 06 10:41:06.625708 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.625640    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551"} err="failed to get container status \"54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551\": rpc error: code = NotFound desc = could not find container \"54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551\": container with ID starting with 54ab85fd8767cd8cc918c497ddafb462d0ac02f43ba766bd264bc578eb63d551 not found: ID does not exist"
Sep 06 10:41:06.637486 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.637448    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"]
Sep 06 10:41:06.641133 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:06.641102    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b21e0db6-b66bf747c-j6pws"]
Sep 06 10:41:07.246020 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:07.245975    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16ac05bd-e877-4919-9331-f88ecaad9875" path="/var/lib/kubelet/pods/16ac05bd-e877-4919-9331-f88ecaad9875/volumes"
Sep 06 10:41:23.071591 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:23.071508    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:41:39.674887 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:39.674851    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:41:56.581086 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:41:56.581044    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:42:12.074113 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:12.074080    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:42:27.368251 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:27.368205    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:42:42.577573 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:42.577533    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:42:43.104856 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:43.104819    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"]
Sep 06 10:42:51.968759 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:51.968727    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:42:55.768272 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.768232    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"]
Sep 06 10:42:55.768812 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.768796    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16ac05bd-e877-4919-9331-f88ecaad9875" containerName="payload-processing"
Sep 06 10:42:55.768865 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.768816    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="16ac05bd-e877-4919-9331-f88ecaad9875" containerName="payload-processing"
Sep 06 10:42:55.768949 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.768937    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="16ac05bd-e877-4919-9331-f88ecaad9875" containerName="payload-processing"
Sep 06 10:42:55.772218 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.772195    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.774265 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.774242    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-616a10-serving-cert\""
Sep 06 10:42:55.784825 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.784788    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"]
Sep 06 10:42:55.846470 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.846438    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-metrics-certs\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.846470 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.846477    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmj7\" (UniqueName: \"kubernetes.io/projected/c84943e8-9624-46ab-8317-4257de886816-kube-api-access-kbmj7\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.846711 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.846616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84943e8-9624-46ab-8317-4257de886816-openshift-service-ca\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.846751 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.846723    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-maas-api-tls\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.947354 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.947310    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-maas-api-tls\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.947525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.947398    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-metrics-certs\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.947525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.947418    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmj7\" (UniqueName: \"kubernetes.io/projected/c84943e8-9624-46ab-8317-4257de886816-kube-api-access-kbmj7\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.947525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.947456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84943e8-9624-46ab-8317-4257de886816-openshift-service-ca\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.947525 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:42:55.947458    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-serving-cert: secret "maas-api-e2e-api-a-616a10-serving-cert" not found
Sep 06 10:42:55.947662 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:42:55.947593    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-maas-api-tls podName:c84943e8-9624-46ab-8317-4257de886816 nodeName:}" failed. No retries permitted until 2026-09-06 10:42:56.447573033 +0000 UTC m=+1267.720214515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-maas-api-tls") pod "maas-api-e2e-api-a-616a10-8678c8687-mlnq8" (UID: "c84943e8-9624-46ab-8317-4257de886816") : secret "maas-api-e2e-api-a-616a10-serving-cert" not found
Sep 06 10:42:55.948185 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.948155    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84943e8-9624-46ab-8317-4257de886816-openshift-service-ca\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.950111 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.950080    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-metrics-certs\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:55.958495 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:55.958460    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmj7\" (UniqueName: \"kubernetes.io/projected/c84943e8-9624-46ab-8317-4257de886816-kube-api-access-kbmj7\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:56.451474 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:56.451439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-maas-api-tls\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:56.453921 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:56.453895    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-maas-api-tls\") pod \"maas-api-e2e-api-a-616a10-8678c8687-mlnq8\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:56.683689 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:56.683647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:56.957616 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:56.957592    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"]
Sep 06 10:42:56.960321 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:42:56.960291    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84943e8_9624_46ab_8317_4257de886816.slice/crio-bb9566ebf8595cb058c6c0f464c7343ffac1a641ab141d002547fc9efd54c7e7 WatchSource:0}: Error finding container bb9566ebf8595cb058c6c0f464c7343ffac1a641ab141d002547fc9efd54c7e7: Status 404 returned error can't find the container with id bb9566ebf8595cb058c6c0f464c7343ffac1a641ab141d002547fc9efd54c7e7
Sep 06 10:42:57.158322 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:57.158285    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8" event={"ID":"c84943e8-9624-46ab-8317-4257de886816","Type":"ContainerStarted","Data":"bb9566ebf8595cb058c6c0f464c7343ffac1a641ab141d002547fc9efd54c7e7"}
Sep 06 10:42:58.126444 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.126397    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6" podUID="6a356caa-cf45-4b8f-a74e-d489750dda80" containerName="main" containerID="cri-o://b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5" gracePeriod=45
Sep 06 10:42:58.164968 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.164930    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8" event={"ID":"c84943e8-9624-46ab-8317-4257de886816","Type":"ContainerStarted","Data":"e525c86ef690298a982ac48368f388788b0d4e271f80a0e4591271e4d8bdaa42"}
Sep 06 10:42:58.165159 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.165042    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:42:58.181243 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.181183    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8" podStartSLOduration=2.800072222 podStartE2EDuration="3.181166133s" totalImagesPullingTime="381.093911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:42:58.180272167 +0000 UTC m=+1269.452913671" watchObservedRunningTime="2026-09-06 10:42:58.181166133 +0000 UTC m=+1269.453807636"
Sep 06 10:42:58.395208 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.395177    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:42:58.470469 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.470423    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-dshm\") pod \"6a356caa-cf45-4b8f-a74e-d489750dda80\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") "
Sep 06 10:42:58.470469 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.470473    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a356caa-cf45-4b8f-a74e-d489750dda80-tls-certs\") pod \"6a356caa-cf45-4b8f-a74e-d489750dda80\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") "
Sep 06 10:42:58.470740 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.470495    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7qln\" (UniqueName: \"kubernetes.io/projected/6a356caa-cf45-4b8f-a74e-d489750dda80-kube-api-access-v7qln\") pod \"6a356caa-cf45-4b8f-a74e-d489750dda80\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") "
Sep 06 10:42:58.470740 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.470563    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-model-cache\") pod \"6a356caa-cf45-4b8f-a74e-d489750dda80\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") "
Sep 06 10:42:58.470740 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.470590    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-home\") pod \"6a356caa-cf45-4b8f-a74e-d489750dda80\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") "
Sep 06 10:42:58.470740 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.470613    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-tmp-dir\") pod \"6a356caa-cf45-4b8f-a74e-d489750dda80\" (UID: \"6a356caa-cf45-4b8f-a74e-d489750dda80\") "
Sep 06 10:42:58.470962 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.470820    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a356caa-cf45-4b8f-a74e-d489750dda80" (UID: "6a356caa-cf45-4b8f-a74e-d489750dda80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:42:58.470962 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.470922    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-home" (OuterVolumeSpecName: "home") pod "6a356caa-cf45-4b8f-a74e-d489750dda80" (UID: "6a356caa-cf45-4b8f-a74e-d489750dda80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:42:58.471060 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.471005    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a356caa-cf45-4b8f-a74e-d489750dda80" (UID: "6a356caa-cf45-4b8f-a74e-d489750dda80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:42:58.472790 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.472760    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-dshm" (OuterVolumeSpecName: "dshm") pod "6a356caa-cf45-4b8f-a74e-d489750dda80" (UID: "6a356caa-cf45-4b8f-a74e-d489750dda80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:42:58.472922 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.472825    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a356caa-cf45-4b8f-a74e-d489750dda80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a356caa-cf45-4b8f-a74e-d489750dda80" (UID: "6a356caa-cf45-4b8f-a74e-d489750dda80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:42:58.472922 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.472849    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a356caa-cf45-4b8f-a74e-d489750dda80-kube-api-access-v7qln" (OuterVolumeSpecName: "kube-api-access-v7qln") pod "6a356caa-cf45-4b8f-a74e-d489750dda80" (UID: "6a356caa-cf45-4b8f-a74e-d489750dda80"). InnerVolumeSpecName "kube-api-access-v7qln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:42:58.571778 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.571722    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7qln\" (UniqueName: \"kubernetes.io/projected/6a356caa-cf45-4b8f-a74e-d489750dda80-kube-api-access-v7qln\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:42:58.571778 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.571769    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-model-cache\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:42:58.571778 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.571778    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-home\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:42:58.571778 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.571786    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-tmp-dir\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:42:58.571778 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.571793    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a356caa-cf45-4b8f-a74e-d489750dda80-dshm\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:42:58.571778 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:58.571800    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a356caa-cf45-4b8f-a74e-d489750dda80-tls-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:42:59.171957 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.171918    2583 generic.go:356] "Generic (PLEG): container finished" podID="6a356caa-cf45-4b8f-a74e-d489750dda80" containerID="b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5" exitCode=0
Sep 06 10:42:59.172474 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.171965    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"
Sep 06 10:42:59.172474 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.172010    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6" event={"ID":"6a356caa-cf45-4b8f-a74e-d489750dda80","Type":"ContainerDied","Data":"b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5"}
Sep 06 10:42:59.172474 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.172068    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6" event={"ID":"6a356caa-cf45-4b8f-a74e-d489750dda80","Type":"ContainerDied","Data":"691c9888cfcb1bd74f87a6d23ce0850a33aa613b2e7bd92d8eb4fea88f495fe9"}
Sep 06 10:42:59.172474 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.172112    2583 scope.go:117] "RemoveContainer" containerID="b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5"
Sep 06 10:42:59.181246 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.181216    2583 scope.go:117] "RemoveContainer" containerID="b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5"
Sep 06 10:42:59.181516 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:42:59.181499    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5\": container with ID starting with b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5 not found: ID does not exist" containerID="b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5"
Sep 06 10:42:59.181576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.181523    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5"} err="failed to get container status \"b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5\": rpc error: code = NotFound desc = could not find container \"b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5\": container with ID starting with b559ee9921faeb797d9401b91fbeb117a2f897252f0e5f3ab2e3bdbf050bd9d5 not found: ID does not exist"
Sep 06 10:42:59.191830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.191801    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"]
Sep 06 10:42:59.194328 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.194290    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/auth-test-model-682ea0-kserve-dcb88cd67-f24s6"]
Sep 06 10:42:59.240626 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:42:59.240588    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a356caa-cf45-4b8f-a74e-d489750dda80" path="/var/lib/kubelet/pods/6a356caa-cf45-4b8f-a74e-d489750dda80/volumes"
Sep 06 10:43:01.579758 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:01.579720    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:43:04.178595 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:04.178566    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:43:16.575899 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:16.575852    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:43:25.073091 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.073056    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"]
Sep 06 10:43:25.073849 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.073826    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a356caa-cf45-4b8f-a74e-d489750dda80" containerName="main"
Sep 06 10:43:25.073919 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.073855    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a356caa-cf45-4b8f-a74e-d489750dda80" containerName="main"
Sep 06 10:43:25.073978 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.073958    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a356caa-cf45-4b8f-a74e-d489750dda80" containerName="main"
Sep 06 10:43:25.077598 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.077572    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.079418 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.079393    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-40f309-serving-cert\""
Sep 06 10:43:25.085930 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.084587    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"]
Sep 06 10:43:25.223087 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.223045    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6lt2\" (UniqueName: \"kubernetes.io/projected/c229f540-e955-4a41-b34b-92e41182c00c-kube-api-access-p6lt2\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.223274 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.223236    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-metrics-certs\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.223274 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.223267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-maas-api-tls\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.223395 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.223291    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c229f540-e955-4a41-b34b-92e41182c00c-openshift-service-ca\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.324496 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.324410    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-metrics-certs\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.324496 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.324450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-maas-api-tls\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.324496 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.324470    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c229f540-e955-4a41-b34b-92e41182c00c-openshift-service-ca\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.324779 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:43:25.324649    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-serving-cert: secret "maas-api-e2e-api-b-40f309-serving-cert" not found
Sep 06 10:43:25.324779 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.324686    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6lt2\" (UniqueName: \"kubernetes.io/projected/c229f540-e955-4a41-b34b-92e41182c00c-kube-api-access-p6lt2\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.324779 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:43:25.324725    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-maas-api-tls podName:c229f540-e955-4a41-b34b-92e41182c00c nodeName:}" failed. No retries permitted until 2026-09-06 10:43:25.824702834 +0000 UTC m=+1297.097344337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-maas-api-tls") pod "maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n" (UID: "c229f540-e955-4a41-b34b-92e41182c00c") : secret "maas-api-e2e-api-b-40f309-serving-cert" not found
Sep 06 10:43:25.325236 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.325205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c229f540-e955-4a41-b34b-92e41182c00c-openshift-service-ca\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.326979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.326958    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-metrics-certs\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.333663 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.333597    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6lt2\" (UniqueName: \"kubernetes.io/projected/c229f540-e955-4a41-b34b-92e41182c00c-kube-api-access-p6lt2\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.829918 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.829881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-maas-api-tls\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.832465 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.832440    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-maas-api-tls\") pod \"maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:25.891805 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.891771    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"]
Sep 06 10:43:25.895540 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.895516    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:25.897858 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.897818    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-40f309\""
Sep 06 10:43:25.898229 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.898212    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-40f309-dockercfg-x4w2f\""
Sep 06 10:43:25.907796 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.907772    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"]
Sep 06 10:43:25.990615 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:25.990577    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:26.031271 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.031048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2z4k\" (UniqueName: \"kubernetes.io/projected/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-kube-api-access-b2z4k\") pod \"payload-processing-e2e-api-b-40f309-d8d77789-msr5h\" (UID: \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:26.031271 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.031146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-40f309-d8d77789-msr5h\" (UID: \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:26.131997 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.131960    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2z4k\" (UniqueName: \"kubernetes.io/projected/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-kube-api-access-b2z4k\") pod \"payload-processing-e2e-api-b-40f309-d8d77789-msr5h\" (UID: \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:26.132401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.132018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-40f309-d8d77789-msr5h\" (UID: \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:26.132673 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.132654    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-40f309-d8d77789-msr5h\" (UID: \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:26.139821 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.139793    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2z4k\" (UniqueName: \"kubernetes.io/projected/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-kube-api-access-b2z4k\") pod \"payload-processing-e2e-api-b-40f309-d8d77789-msr5h\" (UID: \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\") " pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:26.206523 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.206462    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:26.267979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.267894    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"]
Sep 06 10:43:26.272222 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:43:26.272190    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc229f540_e955_4a41_b34b_92e41182c00c.slice/crio-5aeb9f434228e412b3a21e0c88f9ea3eb8a3a221e7fa3a2393acf7086bb9ed14 WatchSource:0}: Error finding container 5aeb9f434228e412b3a21e0c88f9ea3eb8a3a221e7fa3a2393acf7086bb9ed14: Status 404 returned error can't find the container with id 5aeb9f434228e412b3a21e0c88f9ea3eb8a3a221e7fa3a2393acf7086bb9ed14
Sep 06 10:43:26.273781 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.273759    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:43:26.305161 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.305128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n" event={"ID":"c229f540-e955-4a41-b34b-92e41182c00c","Type":"ContainerStarted","Data":"5aeb9f434228e412b3a21e0c88f9ea3eb8a3a221e7fa3a2393acf7086bb9ed14"}
Sep 06 10:43:26.477287 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:26.477240    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"]
Sep 06 10:43:26.479994 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:43:26.479963    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5f1ce5_f54d_4ab9_b413_288fc1597fd3.slice/crio-4b1816334d16671f476603220af93f3f6b9378b9101b29f105bd8e126407987a WatchSource:0}: Error finding container 4b1816334d16671f476603220af93f3f6b9378b9101b29f105bd8e126407987a: Status 404 returned error can't find the container with id 4b1816334d16671f476603220af93f3f6b9378b9101b29f105bd8e126407987a
Sep 06 10:43:27.319301 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:27.319264    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n" event={"ID":"c229f540-e955-4a41-b34b-92e41182c00c","Type":"ContainerStarted","Data":"6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743"}
Sep 06 10:43:27.319801 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:27.319388    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:27.320910 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:27.320882    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h" event={"ID":"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3","Type":"ContainerStarted","Data":"95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5"}
Sep 06 10:43:27.320995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:27.320936    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h" event={"ID":"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3","Type":"ContainerStarted","Data":"4b1816334d16671f476603220af93f3f6b9378b9101b29f105bd8e126407987a"}
Sep 06 10:43:27.321050 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:27.320998    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:27.334760 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:27.334701    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n" podStartSLOduration=1.95691936 podStartE2EDuration="2.334681023s" totalImagesPullingTime="377.761663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:43:27.334589494 +0000 UTC m=+1298.607230997" watchObservedRunningTime="2026-09-06 10:43:27.334681023 +0000 UTC m=+1298.607322530"
Sep 06 10:43:27.351349 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:27.351284    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h" podStartSLOduration=2.351267251 podStartE2EDuration="2.351267251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:43:27.349055655 +0000 UTC m=+1298.621697151" watchObservedRunningTime="2026-09-06 10:43:27.351267251 +0000 UTC m=+1298.623908797"
Sep 06 10:43:32.786403 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:32.786354    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:43:33.330433 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:33.330402    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:43:38.327542 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:38.327510    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:43:46.774639 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:43:46.774598    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:44:01.787578 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:01.787538    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:44:07.303246 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.303209    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"]
Sep 06 10:44:07.303682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.303492    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8" podUID="c84943e8-9624-46ab-8317-4257de886816" containerName="maas-api" containerID="cri-o://e525c86ef690298a982ac48368f388788b0d4e271f80a0e4591271e4d8bdaa42" gracePeriod=30
Sep 06 10:44:07.519803 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.519608    2583 generic.go:356] "Generic (PLEG): container finished" podID="c84943e8-9624-46ab-8317-4257de886816" containerID="e525c86ef690298a982ac48368f388788b0d4e271f80a0e4591271e4d8bdaa42" exitCode=0
Sep 06 10:44:07.519803 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.519731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8" event={"ID":"c84943e8-9624-46ab-8317-4257de886816","Type":"ContainerDied","Data":"e525c86ef690298a982ac48368f388788b0d4e271f80a0e4591271e4d8bdaa42"}
Sep 06 10:44:07.560188 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.560129    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:44:07.665222 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.665188    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-maas-api-tls\") pod \"c84943e8-9624-46ab-8317-4257de886816\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") "
Sep 06 10:44:07.665550 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.665245    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-metrics-certs\") pod \"c84943e8-9624-46ab-8317-4257de886816\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") "
Sep 06 10:44:07.665550 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.665263    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbmj7\" (UniqueName: \"kubernetes.io/projected/c84943e8-9624-46ab-8317-4257de886816-kube-api-access-kbmj7\") pod \"c84943e8-9624-46ab-8317-4257de886816\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") "
Sep 06 10:44:07.665550 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.665303    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84943e8-9624-46ab-8317-4257de886816-openshift-service-ca\") pod \"c84943e8-9624-46ab-8317-4257de886816\" (UID: \"c84943e8-9624-46ab-8317-4257de886816\") "
Sep 06 10:44:07.665832 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.665806    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c84943e8-9624-46ab-8317-4257de886816-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c84943e8-9624-46ab-8317-4257de886816" (UID: "c84943e8-9624-46ab-8317-4257de886816"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:07.667619 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.667594    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c84943e8-9624-46ab-8317-4257de886816" (UID: "c84943e8-9624-46ab-8317-4257de886816"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:07.667911 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.667888    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c84943e8-9624-46ab-8317-4257de886816" (UID: "c84943e8-9624-46ab-8317-4257de886816"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:07.667976 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.667916    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c84943e8-9624-46ab-8317-4257de886816-kube-api-access-kbmj7" (OuterVolumeSpecName: "kube-api-access-kbmj7") pod "c84943e8-9624-46ab-8317-4257de886816" (UID: "c84943e8-9624-46ab-8317-4257de886816"). InnerVolumeSpecName "kube-api-access-kbmj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:07.766518 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.766482    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84943e8-9624-46ab-8317-4257de886816-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:07.766518 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.766512    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:07.766518 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.766521    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c84943e8-9624-46ab-8317-4257de886816-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:07.766518 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:07.766529    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmj7\" (UniqueName: \"kubernetes.io/projected/c84943e8-9624-46ab-8317-4257de886816-kube-api-access-kbmj7\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:08.526119 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:08.526081    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8" event={"ID":"c84943e8-9624-46ab-8317-4257de886816","Type":"ContainerDied","Data":"bb9566ebf8595cb058c6c0f464c7343ffac1a641ab141d002547fc9efd54c7e7"}
Sep 06 10:44:08.526119 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:08.526111    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"
Sep 06 10:44:08.526629 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:08.526158    2583 scope.go:117] "RemoveContainer" containerID="e525c86ef690298a982ac48368f388788b0d4e271f80a0e4591271e4d8bdaa42"
Sep 06 10:44:08.552395 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:08.552355    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"]
Sep 06 10:44:08.558535 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:08.558506    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-616a10-8678c8687-mlnq8"]
Sep 06 10:44:09.240691 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:09.240657    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c84943e8-9624-46ab-8317-4257de886816" path="/var/lib/kubelet/pods/c84943e8-9624-46ab-8317-4257de886816/volumes"
Sep 06 10:44:22.269957 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:22.269874    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:44:29.990320 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:29.989867    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"]
Sep 06 10:44:29.990320 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:29.990113    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n" podUID="c229f540-e955-4a41-b34b-92e41182c00c" containerName="maas-api" containerID="cri-o://6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743" gracePeriod=30
Sep 06 10:44:30.042936 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.042902    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"]
Sep 06 10:44:30.043126 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.043100    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h" podUID="4f5f1ce5-f54d-4ab9-b413-288fc1597fd3" containerName="payload-processing" containerID="cri-o://95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5" gracePeriod=30
Sep 06 10:44:30.293237 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.293189    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:44:30.329560 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.329529    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:44:30.380846 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.380808    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6lt2\" (UniqueName: \"kubernetes.io/projected/c229f540-e955-4a41-b34b-92e41182c00c-kube-api-access-p6lt2\") pod \"c229f540-e955-4a41-b34b-92e41182c00c\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") "
Sep 06 10:44:30.381052 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.380863    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-metrics-certs\") pod \"c229f540-e955-4a41-b34b-92e41182c00c\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") "
Sep 06 10:44:30.381052 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.380978    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-maas-api-tls\") pod \"c229f540-e955-4a41-b34b-92e41182c00c\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") "
Sep 06 10:44:30.381052 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.381042    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c229f540-e955-4a41-b34b-92e41182c00c-openshift-service-ca\") pod \"c229f540-e955-4a41-b34b-92e41182c00c\" (UID: \"c229f540-e955-4a41-b34b-92e41182c00c\") "
Sep 06 10:44:30.381498 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.381470    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c229f540-e955-4a41-b34b-92e41182c00c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c229f540-e955-4a41-b34b-92e41182c00c" (UID: "c229f540-e955-4a41-b34b-92e41182c00c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:30.383008 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.382978    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c229f540-e955-4a41-b34b-92e41182c00c-kube-api-access-p6lt2" (OuterVolumeSpecName: "kube-api-access-p6lt2") pod "c229f540-e955-4a41-b34b-92e41182c00c" (UID: "c229f540-e955-4a41-b34b-92e41182c00c"). InnerVolumeSpecName "kube-api-access-p6lt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:30.383008 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.382995    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c229f540-e955-4a41-b34b-92e41182c00c" (UID: "c229f540-e955-4a41-b34b-92e41182c00c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:30.383127 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.383040    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c229f540-e955-4a41-b34b-92e41182c00c" (UID: "c229f540-e955-4a41-b34b-92e41182c00c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:30.482530 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.482487    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-plugins-config-volume\") pod \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\" (UID: \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\") "
Sep 06 10:44:30.482705 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.482544    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2z4k\" (UniqueName: \"kubernetes.io/projected/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-kube-api-access-b2z4k\") pod \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\" (UID: \"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3\") "
Sep 06 10:44:30.482753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.482704    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6lt2\" (UniqueName: \"kubernetes.io/projected/c229f540-e955-4a41-b34b-92e41182c00c-kube-api-access-p6lt2\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:30.482753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.482715    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:30.482753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.482724    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c229f540-e955-4a41-b34b-92e41182c00c-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:30.482753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.482733    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c229f540-e955-4a41-b34b-92e41182c00c-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:30.482952 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.482924    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4f5f1ce5-f54d-4ab9-b413-288fc1597fd3" (UID: "4f5f1ce5-f54d-4ab9-b413-288fc1597fd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:30.484606 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.484584    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-kube-api-access-b2z4k" (OuterVolumeSpecName: "kube-api-access-b2z4k") pod "4f5f1ce5-f54d-4ab9-b413-288fc1597fd3" (UID: "4f5f1ce5-f54d-4ab9-b413-288fc1597fd3"). InnerVolumeSpecName "kube-api-access-b2z4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:30.583523 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.583435    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:30.583523 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.583465    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2z4k\" (UniqueName: \"kubernetes.io/projected/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3-kube-api-access-b2z4k\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:30.634177 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.634138    2583 generic.go:356] "Generic (PLEG): container finished" podID="c229f540-e955-4a41-b34b-92e41182c00c" containerID="6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743" exitCode=0
Sep 06 10:44:30.634378 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.634187    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"
Sep 06 10:44:30.634378 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.634221    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n" event={"ID":"c229f540-e955-4a41-b34b-92e41182c00c","Type":"ContainerDied","Data":"6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743"}
Sep 06 10:44:30.634378 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.634280    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n" event={"ID":"c229f540-e955-4a41-b34b-92e41182c00c","Type":"ContainerDied","Data":"5aeb9f434228e412b3a21e0c88f9ea3eb8a3a221e7fa3a2393acf7086bb9ed14"}
Sep 06 10:44:30.634378 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.634312    2583 scope.go:117] "RemoveContainer" containerID="6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743"
Sep 06 10:44:30.635899 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.635877    2583 generic.go:356] "Generic (PLEG): container finished" podID="4f5f1ce5-f54d-4ab9-b413-288fc1597fd3" containerID="95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5" exitCode=0
Sep 06 10:44:30.636014 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.635926    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"
Sep 06 10:44:30.636014 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.635952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h" event={"ID":"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3","Type":"ContainerDied","Data":"95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5"}
Sep 06 10:44:30.636014 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.635985    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h" event={"ID":"4f5f1ce5-f54d-4ab9-b413-288fc1597fd3","Type":"ContainerDied","Data":"4b1816334d16671f476603220af93f3f6b9378b9101b29f105bd8e126407987a"}
Sep 06 10:44:30.643128 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.643100    2583 scope.go:117] "RemoveContainer" containerID="6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743"
Sep 06 10:44:30.643479 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:44:30.643456    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743\": container with ID starting with 6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743 not found: ID does not exist" containerID="6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743"
Sep 06 10:44:30.643562 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.643490    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743"} err="failed to get container status \"6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743\": rpc error: code = NotFound desc = could not find container \"6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743\": container with ID starting with 6d2c6cb3ea1166082772fa4d27e754112223539ef9ff18bfe1df9d0ddd084743 not found: ID does not exist"
Sep 06 10:44:30.643562 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.643506    2583 scope.go:117] "RemoveContainer" containerID="95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5"
Sep 06 10:44:30.651732 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.651714    2583 scope.go:117] "RemoveContainer" containerID="95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5"
Sep 06 10:44:30.652021 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:44:30.652002    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5\": container with ID starting with 95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5 not found: ID does not exist" containerID="95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5"
Sep 06 10:44:30.652076 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.652032    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5"} err="failed to get container status \"95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5\": rpc error: code = NotFound desc = could not find container \"95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5\": container with ID starting with 95f63a0d52409f4d1f722d9ea4e0f981e3ce5812e447d2318b481ea6e3567fc5 not found: ID does not exist"
Sep 06 10:44:30.659525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.659496    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"]
Sep 06 10:44:30.662952 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.662927    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-40f309-5fcbd8d6d9-w225n"]
Sep 06 10:44:30.672515 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.672488    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"]
Sep 06 10:44:30.675800 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:30.675771    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-40f309-d8d77789-msr5h"]
Sep 06 10:44:31.242130 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:31.242097    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f5f1ce5-f54d-4ab9-b413-288fc1597fd3" path="/var/lib/kubelet/pods/4f5f1ce5-f54d-4ab9-b413-288fc1597fd3/volumes"
Sep 06 10:44:31.242513 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:31.242505    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c229f540-e955-4a41-b34b-92e41182c00c" path="/var/lib/kubelet/pods/c229f540-e955-4a41-b34b-92e41182c00c/volumes"
Sep 06 10:44:32.872156 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:32.872117    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:44:47.429838 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.429802    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"]
Sep 06 10:44:47.430721 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.430688    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8" podUID="218e9039-ceee-4386-aed4-96a1e3f71d2b" containerName="maas-api" containerID="cri-o://fb3f7bbe2bed655864652ecbca9f87e6c9c0031c3870019bebe34cf3b839a0ae" gracePeriod=30
Sep 06 10:44:47.479646 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.479611    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"]
Sep 06 10:44:47.479888 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.479861    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc" podUID="4837282d-e2ac-40b2-a8cf-d311cbe18584" containerName="payload-processing" containerID="cri-o://fa3640b307663c74dfa39ca674e7796eb5d01c0939edba1b2fce745bb4710c1a" gracePeriod=30
Sep 06 10:44:47.723108 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.723072    2583 generic.go:356] "Generic (PLEG): container finished" podID="4837282d-e2ac-40b2-a8cf-d311cbe18584" containerID="fa3640b307663c74dfa39ca674e7796eb5d01c0939edba1b2fce745bb4710c1a" exitCode=0
Sep 06 10:44:47.723255 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.723146    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc" event={"ID":"4837282d-e2ac-40b2-a8cf-d311cbe18584","Type":"ContainerDied","Data":"fa3640b307663c74dfa39ca674e7796eb5d01c0939edba1b2fce745bb4710c1a"}
Sep 06 10:44:47.724811 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.724785    2583 generic.go:356] "Generic (PLEG): container finished" podID="218e9039-ceee-4386-aed4-96a1e3f71d2b" containerID="fb3f7bbe2bed655864652ecbca9f87e6c9c0031c3870019bebe34cf3b839a0ae" exitCode=0
Sep 06 10:44:47.724918 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.724858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8" event={"ID":"218e9039-ceee-4386-aed4-96a1e3f71d2b","Type":"ContainerDied","Data":"fb3f7bbe2bed655864652ecbca9f87e6c9c0031c3870019bebe34cf3b839a0ae"}
Sep 06 10:44:47.724918 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.724911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8" event={"ID":"218e9039-ceee-4386-aed4-96a1e3f71d2b","Type":"ContainerDied","Data":"7c738c14a2390a40e0e82051b86fc62c6dd83e276c15ca889e06bf507867ab2d"}
Sep 06 10:44:47.724988 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.724945    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c738c14a2390a40e0e82051b86fc62c6dd83e276c15ca889e06bf507867ab2d"
Sep 06 10:44:47.736302 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.736090    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:44:47.771468 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.771444    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:44:47.832108 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.832069    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkwmr\" (UniqueName: \"kubernetes.io/projected/218e9039-ceee-4386-aed4-96a1e3f71d2b-kube-api-access-wkwmr\") pod \"218e9039-ceee-4386-aed4-96a1e3f71d2b\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") "
Sep 06 10:44:47.832277 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.832133    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-maas-api-tls\") pod \"218e9039-ceee-4386-aed4-96a1e3f71d2b\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") "
Sep 06 10:44:47.832277 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.832166    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bnb4\" (UniqueName: \"kubernetes.io/projected/4837282d-e2ac-40b2-a8cf-d311cbe18584-kube-api-access-2bnb4\") pod \"4837282d-e2ac-40b2-a8cf-d311cbe18584\" (UID: \"4837282d-e2ac-40b2-a8cf-d311cbe18584\") "
Sep 06 10:44:47.832277 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.832244    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218e9039-ceee-4386-aed4-96a1e3f71d2b-openshift-service-ca\") pod \"218e9039-ceee-4386-aed4-96a1e3f71d2b\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") "
Sep 06 10:44:47.832464 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.832294    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-metrics-certs\") pod \"218e9039-ceee-4386-aed4-96a1e3f71d2b\" (UID: \"218e9039-ceee-4386-aed4-96a1e3f71d2b\") "
Sep 06 10:44:47.832464 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.832348    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4837282d-e2ac-40b2-a8cf-d311cbe18584-plugins-config-volume\") pod \"4837282d-e2ac-40b2-a8cf-d311cbe18584\" (UID: \"4837282d-e2ac-40b2-a8cf-d311cbe18584\") "
Sep 06 10:44:47.832816 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.832702    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218e9039-ceee-4386-aed4-96a1e3f71d2b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "218e9039-ceee-4386-aed4-96a1e3f71d2b" (UID: "218e9039-ceee-4386-aed4-96a1e3f71d2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:47.832916 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.832886    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4837282d-e2ac-40b2-a8cf-d311cbe18584-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4837282d-e2ac-40b2-a8cf-d311cbe18584" (UID: "4837282d-e2ac-40b2-a8cf-d311cbe18584"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:47.834479 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.834450    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218e9039-ceee-4386-aed4-96a1e3f71d2b-kube-api-access-wkwmr" (OuterVolumeSpecName: "kube-api-access-wkwmr") pod "218e9039-ceee-4386-aed4-96a1e3f71d2b" (UID: "218e9039-ceee-4386-aed4-96a1e3f71d2b"). InnerVolumeSpecName "kube-api-access-wkwmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:47.834790 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.834772    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "218e9039-ceee-4386-aed4-96a1e3f71d2b" (UID: "218e9039-ceee-4386-aed4-96a1e3f71d2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:47.834857 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.834826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "218e9039-ceee-4386-aed4-96a1e3f71d2b" (UID: "218e9039-ceee-4386-aed4-96a1e3f71d2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:47.834857 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.834835    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4837282d-e2ac-40b2-a8cf-d311cbe18584-kube-api-access-2bnb4" (OuterVolumeSpecName: "kube-api-access-2bnb4") pod "4837282d-e2ac-40b2-a8cf-d311cbe18584" (UID: "4837282d-e2ac-40b2-a8cf-d311cbe18584"). InnerVolumeSpecName "kube-api-access-2bnb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:47.933864 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.933829    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/218e9039-ceee-4386-aed4-96a1e3f71d2b-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:47.933864 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.933859    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:47.933864 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.933867    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4837282d-e2ac-40b2-a8cf-d311cbe18584-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:47.934082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.933877    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkwmr\" (UniqueName: \"kubernetes.io/projected/218e9039-ceee-4386-aed4-96a1e3f71d2b-kube-api-access-wkwmr\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:47.934082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.933886    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/218e9039-ceee-4386-aed4-96a1e3f71d2b-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:47.934082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:47.933893    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bnb4\" (UniqueName: \"kubernetes.io/projected/4837282d-e2ac-40b2-a8cf-d311cbe18584-kube-api-access-2bnb4\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:44:48.731110 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:48.731025    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"
Sep 06 10:44:48.731110 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:48.731063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc" event={"ID":"4837282d-e2ac-40b2-a8cf-d311cbe18584","Type":"ContainerDied","Data":"4bdd0807227bdeb7c02b9bc810dc0604daf603f657978b53a0db612cfa8889a9"}
Sep 06 10:44:48.731613 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:48.731130    2583 scope.go:117] "RemoveContainer" containerID="fa3640b307663c74dfa39ca674e7796eb5d01c0939edba1b2fce745bb4710c1a"
Sep 06 10:44:48.731613 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:48.731275    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"
Sep 06 10:44:48.754289 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:48.754260    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"]
Sep 06 10:44:48.758278 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:48.758251    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-27a2b9-677fb446f9-qmjxc"]
Sep 06 10:44:48.766888 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:48.766866    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"]
Sep 06 10:44:48.769367 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:48.769320    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-27a2b9-bc47958c4-vbfm8"]
Sep 06 10:44:49.239949 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:49.239919    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="218e9039-ceee-4386-aed4-96a1e3f71d2b" path="/var/lib/kubelet/pods/218e9039-ceee-4386-aed4-96a1e3f71d2b/volumes"
Sep 06 10:44:49.240275 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:49.240257    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4837282d-e2ac-40b2-a8cf-d311cbe18584" path="/var/lib/kubelet/pods/4837282d-e2ac-40b2-a8cf-d311cbe18584/volumes"
Sep 06 10:44:49.361590 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:49.361559    2583 scope.go:117] "RemoveContainer" containerID="fb3f7bbe2bed655864652ecbca9f87e6c9c0031c3870019bebe34cf3b839a0ae"
Sep 06 10:44:51.078750 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:44:51.078713    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:45:00.137640 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.137606    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx"]
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.137993    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f5f1ce5-f54d-4ab9-b413-288fc1597fd3" containerName="payload-processing"
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138006    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f5f1ce5-f54d-4ab9-b413-288fc1597fd3" containerName="payload-processing"
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138013    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c84943e8-9624-46ab-8317-4257de886816" containerName="maas-api"
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138019    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c84943e8-9624-46ab-8317-4257de886816" containerName="maas-api"
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138025    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c229f540-e955-4a41-b34b-92e41182c00c" containerName="maas-api"
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138031    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c229f540-e955-4a41-b34b-92e41182c00c" containerName="maas-api"
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138037    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4837282d-e2ac-40b2-a8cf-d311cbe18584" containerName="payload-processing"
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138042    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4837282d-e2ac-40b2-a8cf-d311cbe18584" containerName="payload-processing"
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138063    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="218e9039-ceee-4386-aed4-96a1e3f71d2b" containerName="maas-api"
Sep 06 10:45:00.138095 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138069    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="218e9039-ceee-4386-aed4-96a1e3f71d2b" containerName="maas-api"
Sep 06 10:45:00.138422 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138133    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="218e9039-ceee-4386-aed4-96a1e3f71d2b" containerName="maas-api"
Sep 06 10:45:00.138422 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138141    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c229f540-e955-4a41-b34b-92e41182c00c" containerName="maas-api"
Sep 06 10:45:00.138422 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138148    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f5f1ce5-f54d-4ab9-b413-288fc1597fd3" containerName="payload-processing"
Sep 06 10:45:00.138422 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138155    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4837282d-e2ac-40b2-a8cf-d311cbe18584" containerName="payload-processing"
Sep 06 10:45:00.138422 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.138161    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c84943e8-9624-46ab-8317-4257de886816" containerName="maas-api"
Sep 06 10:45:00.143128 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.143109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx"
Sep 06 10:45:00.145261 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.145234    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n5b2f\""
Sep 06 10:45:00.148961 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.148908    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx"]
Sep 06 10:45:00.156190 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.156166    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx"
Sep 06 10:45:00.435325 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.435295    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx"]
Sep 06 10:45:00.437825 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:45:00.437797    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd033c207_9a1e_4074_87fa_0fa93f6e005f.slice/crio-349c31ecc3525a120ceffe9875eef159608e4f89b967688a1ab1765d15371e3d WatchSource:0}: Error finding container 349c31ecc3525a120ceffe9875eef159608e4f89b967688a1ab1765d15371e3d: Status 404 returned error can't find the container with id 349c31ecc3525a120ceffe9875eef159608e4f89b967688a1ab1765d15371e3d
Sep 06 10:45:00.791472 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:00.791390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx" event={"ID":"d033c207-9a1e-4074-87fa-0fa93f6e005f","Type":"ContainerStarted","Data":"349c31ecc3525a120ceffe9875eef159608e4f89b967688a1ab1765d15371e3d"}
Sep 06 10:45:04.996393 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:04.996085    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"]
Sep 06 10:45:04.996926 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:04.996317    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw" podUID="8e5bef5e-4131-452a-bb36-0bb0246a703d" containerName="payload-processing" containerID="cri-o://817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175" gracePeriod=30
Sep 06 10:45:05.005644 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.005613    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"]
Sep 06 10:45:05.006091 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.005816    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t" podUID="5cae6352-40a3-4c6b-a0af-b43d01ec3e42" containerName="payload-pre-processing" containerID="cri-o://a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9" gracePeriod=30
Sep 06 10:45:05.534054 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.534025    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:45:05.574614 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.574592    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:45:05.594912 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.594883    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtcv4\" (UniqueName: \"kubernetes.io/projected/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-kube-api-access-gtcv4\") pod \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\" (UID: \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\") "
Sep 06 10:45:05.595079 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.594952    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-plugins-config-volume\") pod \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\" (UID: \"5cae6352-40a3-4c6b-a0af-b43d01ec3e42\") "
Sep 06 10:45:05.595387 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.595357    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cae6352-40a3-4c6b-a0af-b43d01ec3e42" (UID: "5cae6352-40a3-4c6b-a0af-b43d01ec3e42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:45:05.597210 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.597173    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-kube-api-access-gtcv4" (OuterVolumeSpecName: "kube-api-access-gtcv4") pod "5cae6352-40a3-4c6b-a0af-b43d01ec3e42" (UID: "5cae6352-40a3-4c6b-a0af-b43d01ec3e42"). InnerVolumeSpecName "kube-api-access-gtcv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:45:05.695773 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.695688    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zh4v\" (UniqueName: \"kubernetes.io/projected/8e5bef5e-4131-452a-bb36-0bb0246a703d-kube-api-access-8zh4v\") pod \"8e5bef5e-4131-452a-bb36-0bb0246a703d\" (UID: \"8e5bef5e-4131-452a-bb36-0bb0246a703d\") "
Sep 06 10:45:05.695773 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.695726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5bef5e-4131-452a-bb36-0bb0246a703d-plugins-config-volume\") pod \"8e5bef5e-4131-452a-bb36-0bb0246a703d\" (UID: \"8e5bef5e-4131-452a-bb36-0bb0246a703d\") "
Sep 06 10:45:05.695972 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.695955    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtcv4\" (UniqueName: \"kubernetes.io/projected/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-kube-api-access-gtcv4\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:45:05.696014 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.695979    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cae6352-40a3-4c6b-a0af-b43d01ec3e42-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:45:05.696150 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.696129    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5bef5e-4131-452a-bb36-0bb0246a703d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e5bef5e-4131-452a-bb36-0bb0246a703d" (UID: "8e5bef5e-4131-452a-bb36-0bb0246a703d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:45:05.718885 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.718848    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5bef5e-4131-452a-bb36-0bb0246a703d-kube-api-access-8zh4v" (OuterVolumeSpecName: "kube-api-access-8zh4v") pod "8e5bef5e-4131-452a-bb36-0bb0246a703d" (UID: "8e5bef5e-4131-452a-bb36-0bb0246a703d"). InnerVolumeSpecName "kube-api-access-8zh4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:45:05.797260 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.797223    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zh4v\" (UniqueName: \"kubernetes.io/projected/8e5bef5e-4131-452a-bb36-0bb0246a703d-kube-api-access-8zh4v\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:45:05.797260 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.797254    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5bef5e-4131-452a-bb36-0bb0246a703d-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:45:05.817490 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.817458    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx" event={"ID":"d033c207-9a1e-4074-87fa-0fa93f6e005f","Type":"ContainerStarted","Data":"a2951adad88b1bad8e8ccfa0bb6026c224071ccafe7590e47c941e99b1c1cd98"}
Sep 06 10:45:05.818918 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.818891    2583 generic.go:356] "Generic (PLEG): container finished" podID="8e5bef5e-4131-452a-bb36-0bb0246a703d" containerID="817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175" exitCode=0
Sep 06 10:45:05.819056 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.818941    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"
Sep 06 10:45:05.819056 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.818939    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw" event={"ID":"8e5bef5e-4131-452a-bb36-0bb0246a703d","Type":"ContainerDied","Data":"817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175"}
Sep 06 10:45:05.819184 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.819102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw" event={"ID":"8e5bef5e-4131-452a-bb36-0bb0246a703d","Type":"ContainerDied","Data":"48ae3542a1cf37fa29e40e876d6ccbee7ac0f81e7d186236c769b180f2edea6f"}
Sep 06 10:45:05.819184 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.819146    2583 scope.go:117] "RemoveContainer" containerID="817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175"
Sep 06 10:45:05.820647 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.820625    2583 generic.go:356] "Generic (PLEG): container finished" podID="5cae6352-40a3-4c6b-a0af-b43d01ec3e42" containerID="a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9" exitCode=0
Sep 06 10:45:05.820742 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.820710    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t" event={"ID":"5cae6352-40a3-4c6b-a0af-b43d01ec3e42","Type":"ContainerDied","Data":"a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9"}
Sep 06 10:45:05.820742 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.820722    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"
Sep 06 10:45:05.820850 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.820767    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t" event={"ID":"5cae6352-40a3-4c6b-a0af-b43d01ec3e42","Type":"ContainerDied","Data":"2aa64cdf9aa857d42dce3b16188323ceb0f87397f287e1bbf91d440210f6624e"}
Sep 06 10:45:05.829229 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.829201    2583 scope.go:117] "RemoveContainer" containerID="817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175"
Sep 06 10:45:05.829646 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:45:05.829624    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175\": container with ID starting with 817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175 not found: ID does not exist" containerID="817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175"
Sep 06 10:45:05.829734 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.829657    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175"} err="failed to get container status \"817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175\": rpc error: code = NotFound desc = could not find container \"817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175\": container with ID starting with 817e0928d8b5ec17847f95150f705f13fc2efee63a5b02f9e231dd6ec799a175 not found: ID does not exist"
Sep 06 10:45:05.829734 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.829679    2583 scope.go:117] "RemoveContainer" containerID="a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9"
Sep 06 10:45:05.833657 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.833604    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx" podStartSLOduration=0.807233868 podStartE2EDuration="5.833585317s" totalImagesPullingTime="5.026351449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:45:05.831832378 +0000 UTC m=+1397.104473882" watchObservedRunningTime="2026-09-06 10:45:05.833585317 +0000 UTC m=+1397.106226822"
Sep 06 10:45:05.838828 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.838786    2583 scope.go:117] "RemoveContainer" containerID="a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9"
Sep 06 10:45:05.839115 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:45:05.839093    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9\": container with ID starting with a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9 not found: ID does not exist" containerID="a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9"
Sep 06 10:45:05.839214 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.839120    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9"} err="failed to get container status \"a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9\": rpc error: code = NotFound desc = could not find container \"a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9\": container with ID starting with a1437f1b39478d41a5a65d24b3a4065945051d7279ed03de0cae9e14cd5908e9 not found: ID does not exist"
Sep 06 10:45:05.851205 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.851175    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"]
Sep 06 10:45:05.859614 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.859584    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-71e210-588b6d7c4b-ltf6t"]
Sep 06 10:45:05.869404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.869371    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"]
Sep 06 10:45:05.875655 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:05.875631    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-71e210-7db9f8c87f-xvtdw"]
Sep 06 10:45:06.827076 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:06.827040    2583 generic.go:356] "Generic (PLEG): container finished" podID="d033c207-9a1e-4074-87fa-0fa93f6e005f" containerID="a2951adad88b1bad8e8ccfa0bb6026c224071ccafe7590e47c941e99b1c1cd98" exitCode=0
Sep 06 10:45:06.827541 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:06.827120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx" event={"ID":"d033c207-9a1e-4074-87fa-0fa93f6e005f","Type":"ContainerDied","Data":"a2951adad88b1bad8e8ccfa0bb6026c224071ccafe7590e47c941e99b1c1cd98"}
Sep 06 10:45:07.241605 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:07.241572    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cae6352-40a3-4c6b-a0af-b43d01ec3e42" path="/var/lib/kubelet/pods/5cae6352-40a3-4c6b-a0af-b43d01ec3e42/volumes"
Sep 06 10:45:07.241928 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:07.241916    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e5bef5e-4131-452a-bb36-0bb0246a703d" path="/var/lib/kubelet/pods/8e5bef5e-4131-452a-bb36-0bb0246a703d/volumes"
Sep 06 10:45:07.966307 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:07.966278    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx"
Sep 06 10:45:08.070966 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:08.070931    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:45:08.840101 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:08.840069    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx"
Sep 06 10:45:08.840101 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:08.840086    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx" event={"ID":"d033c207-9a1e-4074-87fa-0fa93f6e005f","Type":"ContainerDied","Data":"349c31ecc3525a120ceffe9875eef159608e4f89b967688a1ab1765d15371e3d"}
Sep 06 10:45:08.840311 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:08.840134    2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="349c31ecc3525a120ceffe9875eef159608e4f89b967688a1ab1765d15371e3d"
Sep 06 10:45:31.705576 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705536    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"]
Sep 06 10:45:31.705995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705875    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cae6352-40a3-4c6b-a0af-b43d01ec3e42" containerName="payload-pre-processing"
Sep 06 10:45:31.705995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705886    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cae6352-40a3-4c6b-a0af-b43d01ec3e42" containerName="payload-pre-processing"
Sep 06 10:45:31.705995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705893    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d033c207-9a1e-4074-87fa-0fa93f6e005f" containerName="cleanup"
Sep 06 10:45:31.705995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705899    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d033c207-9a1e-4074-87fa-0fa93f6e005f" containerName="cleanup"
Sep 06 10:45:31.705995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705918    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e5bef5e-4131-452a-bb36-0bb0246a703d" containerName="payload-processing"
Sep 06 10:45:31.705995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705924    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e5bef5e-4131-452a-bb36-0bb0246a703d" containerName="payload-processing"
Sep 06 10:45:31.705995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705978    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e5bef5e-4131-452a-bb36-0bb0246a703d" containerName="payload-processing"
Sep 06 10:45:31.705995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705985    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d033c207-9a1e-4074-87fa-0fa93f6e005f" containerName="cleanup"
Sep 06 10:45:31.705995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.705991    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cae6352-40a3-4c6b-a0af-b43d01ec3e42" containerName="payload-pre-processing"
Sep 06 10:45:31.714047 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.714020    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.716987 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.716965    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"openshift-service-ca.crt\""
Sep 06 10:45:31.717293 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.717245    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"kube-root-ca.crt\""
Sep 06 10:45:31.717472 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.717291    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"default-dockercfg-p7bg2\""
Sep 06 10:45:31.717472 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.717298    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-682ea0\"/\"rate-test-model-682ea0-kserve-self-signed-certs\""
Sep 06 10:45:31.720104 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.720081    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"]
Sep 06 10:45:31.727146 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.727119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-dshm\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.727264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.727150    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g24wr\" (UniqueName: \"kubernetes.io/projected/1b87e5da-2a88-4139-ab98-36e6a1279307-kube-api-access-g24wr\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.727264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.727217    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-home\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.727264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.727252    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-tmp-dir\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.727467 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.727327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-model-cache\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.727467 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.727374    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b87e5da-2a88-4139-ab98-36e6a1279307-tls-certs\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.827742 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.827705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-home\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.827932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.827753    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-tmp-dir\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.827932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.827788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-model-cache\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.827932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.827805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b87e5da-2a88-4139-ab98-36e6a1279307-tls-certs\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.827932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.827825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-dshm\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.827932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.827842    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g24wr\" (UniqueName: \"kubernetes.io/projected/1b87e5da-2a88-4139-ab98-36e6a1279307-kube-api-access-g24wr\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.828253 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.828169    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-home\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.828253 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.828181    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-model-cache\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.828253 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.828241    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-tmp-dir\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.830201 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.830179    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-dshm\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.830491 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.830472    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b87e5da-2a88-4139-ab98-36e6a1279307-tls-certs\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:31.835988 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:31.835956    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g24wr\" (UniqueName: \"kubernetes.io/projected/1b87e5da-2a88-4139-ab98-36e6a1279307-kube-api-access-g24wr\") pod \"rate-test-model-682ea0-kserve-5f96d45459-rvp57\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") " pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:32.025270 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:32.025169    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:32.302132 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:32.302085    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"]
Sep 06 10:45:32.304384 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:45:32.304353    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b87e5da_2a88_4139_ab98_36e6a1279307.slice/crio-63263f2b66f2b8dff3798defcf424b35cce53f03a1f839ccfd88c2dc0e757b17 WatchSource:0}: Error finding container 63263f2b66f2b8dff3798defcf424b35cce53f03a1f839ccfd88c2dc0e757b17: Status 404 returned error can't find the container with id 63263f2b66f2b8dff3798defcf424b35cce53f03a1f839ccfd88c2dc0e757b17
Sep 06 10:45:32.948772 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:32.948738    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57" event={"ID":"1b87e5da-2a88-4139-ab98-36e6a1279307","Type":"ContainerStarted","Data":"3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4"}
Sep 06 10:45:32.948772 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:32.948779    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57" event={"ID":"1b87e5da-2a88-4139-ab98-36e6a1279307","Type":"ContainerStarted","Data":"63263f2b66f2b8dff3798defcf424b35cce53f03a1f839ccfd88c2dc0e757b17"}
Sep 06 10:45:32.965464 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:32.965414    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57" podStartSLOduration=1.9653960270000002 podStartE2EDuration="1.965396027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:45:32.964752072 +0000 UTC m=+1424.237393576" watchObservedRunningTime="2026-09-06 10:45:32.965396027 +0000 UTC m=+1424.238037529"
Sep 06 10:45:35.474012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:35.473974    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:45:42.025801 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:42.025711    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:42.025801 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:42.025775    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:42.038556 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:42.038528    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:45:43.005452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:45:43.005421    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:46:00.072164 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:00.072117    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:46:09.573033 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:09.572981    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:46:15.792967 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:15.792928    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"]
Sep 06 10:46:20.687835 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:20.687793    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:46:21.175887 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.175853    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"]
Sep 06 10:46:21.179149 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.179128    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.181268 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.181248    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7d2fed-serving-cert\""
Sep 06 10:46:21.190504 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.190481    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"]
Sep 06 10:46:21.258989 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.258945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b54c23-2c58-4352-8eb0-e3219d5d82a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.259174 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.259090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.259174 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.259117    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-metrics-certs\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.259174 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.259143    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7fj6\" (UniqueName: \"kubernetes.io/projected/35b54c23-2c58-4352-8eb0-e3219d5d82a4-kube-api-access-k7fj6\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.360553 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.360512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.360742 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.360563    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-metrics-certs\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.360742 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:46:21.360686    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-serving-cert: secret "maas-api-e2e-ipp-a-7d2fed-serving-cert" not found
Sep 06 10:46:21.360865 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:46:21.360766    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-maas-api-tls podName:35b54c23-2c58-4352-8eb0-e3219d5d82a4 nodeName:}" failed. No retries permitted until 2026-09-06 10:46:21.86074148 +0000 UTC m=+1473.133382975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-maas-api-tls") pod "maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4" (UID: "35b54c23-2c58-4352-8eb0-e3219d5d82a4") : secret "maas-api-e2e-ipp-a-7d2fed-serving-cert" not found
Sep 06 10:46:21.361045 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.361018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7fj6\" (UniqueName: \"kubernetes.io/projected/35b54c23-2c58-4352-8eb0-e3219d5d82a4-kube-api-access-k7fj6\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.361129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.361106    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b54c23-2c58-4352-8eb0-e3219d5d82a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.361798 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.361772    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b54c23-2c58-4352-8eb0-e3219d5d82a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.363575 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.363555    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-metrics-certs\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.368503 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.368478    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7fj6\" (UniqueName: \"kubernetes.io/projected/35b54c23-2c58-4352-8eb0-e3219d5d82a4-kube-api-access-k7fj6\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.865752 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.865712    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.868293 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.868264    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:21.924081 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.924046    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"]
Sep 06 10:46:21.927651 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.927629    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:21.929685 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.929660    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7d2fed\""
Sep 06 10:46:21.929819 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.929799    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7d2fed-dockercfg-6qn75\""
Sep 06 10:46:21.936981 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.936957    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"]
Sep 06 10:46:21.966727 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.966692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn\" (UID: \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:21.966915 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:21.966750    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtxz2\" (UniqueName: \"kubernetes.io/projected/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-kube-api-access-vtxz2\") pod \"payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn\" (UID: \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:22.067777 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:22.067728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn\" (UID: \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:22.067959 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:22.067794    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtxz2\" (UniqueName: \"kubernetes.io/projected/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-kube-api-access-vtxz2\") pod \"payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn\" (UID: \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:22.068499 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:22.068480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn\" (UID: \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:22.075260 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:22.075230    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtxz2\" (UniqueName: \"kubernetes.io/projected/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-kube-api-access-vtxz2\") pod \"payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn\" (UID: \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:22.090135 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:22.090109    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:22.239793 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:22.239761    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:22.361391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:22.361366    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"]
Sep 06 10:46:22.363097 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:46:22.363061    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b54c23_2c58_4352_8eb0_e3219d5d82a4.slice/crio-a57c6e192580f317f3674569116a71c219a7f77b84cc78cfccd8d9d10ae4d840 WatchSource:0}: Error finding container a57c6e192580f317f3674569116a71c219a7f77b84cc78cfccd8d9d10ae4d840: Status 404 returned error can't find the container with id a57c6e192580f317f3674569116a71c219a7f77b84cc78cfccd8d9d10ae4d840
Sep 06 10:46:22.509848 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:22.509772    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"]
Sep 06 10:46:22.512976 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:46:22.512947    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f5d411_1de5_4ade_986f_1d59dc69bd5d.slice/crio-457cb8fcaca48ee96b5523ae51e4d6f8db7ef0b589a6e9127a00f3548a0d72fb WatchSource:0}: Error finding container 457cb8fcaca48ee96b5523ae51e4d6f8db7ef0b589a6e9127a00f3548a0d72fb: Status 404 returned error can't find the container with id 457cb8fcaca48ee96b5523ae51e4d6f8db7ef0b589a6e9127a00f3548a0d72fb
Sep 06 10:46:23.183090 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:23.183056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4" event={"ID":"35b54c23-2c58-4352-8eb0-e3219d5d82a4","Type":"ContainerStarted","Data":"561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16"}
Sep 06 10:46:23.183604 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:23.183110    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4" event={"ID":"35b54c23-2c58-4352-8eb0-e3219d5d82a4","Type":"ContainerStarted","Data":"a57c6e192580f317f3674569116a71c219a7f77b84cc78cfccd8d9d10ae4d840"}
Sep 06 10:46:23.183604 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:23.183164    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:23.184695 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:23.184675    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn" event={"ID":"d8f5d411-1de5-4ade-986f-1d59dc69bd5d","Type":"ContainerStarted","Data":"00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc"}
Sep 06 10:46:23.184794 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:23.184714    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn" event={"ID":"d8f5d411-1de5-4ade-986f-1d59dc69bd5d","Type":"ContainerStarted","Data":"457cb8fcaca48ee96b5523ae51e4d6f8db7ef0b589a6e9127a00f3548a0d72fb"}
Sep 06 10:46:23.184845 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:23.184796    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:23.200744 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:23.200700    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4" podStartSLOduration=1.757481715 podStartE2EDuration="2.200685656s" totalImagesPullingTime="443.203941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:23.198572402 +0000 UTC m=+1474.471213902" watchObservedRunningTime="2026-09-06 10:46:23.200685656 +0000 UTC m=+1474.473327158"
Sep 06 10:46:23.214279 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:23.214237    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn" podStartSLOduration=2.214219208 podStartE2EDuration="2.214219208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:23.21352705 +0000 UTC m=+1474.486168554" watchObservedRunningTime="2026-09-06 10:46:23.214219208 +0000 UTC m=+1474.486860763"
Sep 06 10:46:29.195388 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:29.195350    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:46:30.814177 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:30.814131    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57" podUID="1b87e5da-2a88-4139-ab98-36e6a1279307" containerName="main" containerID="cri-o://3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4" gracePeriod=45
Sep 06 10:46:31.068994 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.068937    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:46:31.149659 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.149627    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-home\") pod \"1b87e5da-2a88-4139-ab98-36e6a1279307\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") "
Sep 06 10:46:31.149843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.149688    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-dshm\") pod \"1b87e5da-2a88-4139-ab98-36e6a1279307\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") "
Sep 06 10:46:31.149843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.149726    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-model-cache\") pod \"1b87e5da-2a88-4139-ab98-36e6a1279307\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") "
Sep 06 10:46:31.149843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.149769    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-tmp-dir\") pod \"1b87e5da-2a88-4139-ab98-36e6a1279307\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") "
Sep 06 10:46:31.149843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.149790    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g24wr\" (UniqueName: \"kubernetes.io/projected/1b87e5da-2a88-4139-ab98-36e6a1279307-kube-api-access-g24wr\") pod \"1b87e5da-2a88-4139-ab98-36e6a1279307\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") "
Sep 06 10:46:31.149843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.149816    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b87e5da-2a88-4139-ab98-36e6a1279307-tls-certs\") pod \"1b87e5da-2a88-4139-ab98-36e6a1279307\" (UID: \"1b87e5da-2a88-4139-ab98-36e6a1279307\") "
Sep 06 10:46:31.150106 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.149970    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-home" (OuterVolumeSpecName: "home") pod "1b87e5da-2a88-4139-ab98-36e6a1279307" (UID: "1b87e5da-2a88-4139-ab98-36e6a1279307"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:46:31.150106 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.150059    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b87e5da-2a88-4139-ab98-36e6a1279307" (UID: "1b87e5da-2a88-4139-ab98-36e6a1279307"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:46:31.150106 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.150072    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b87e5da-2a88-4139-ab98-36e6a1279307" (UID: "1b87e5da-2a88-4139-ab98-36e6a1279307"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:46:31.151973 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.151949    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-dshm" (OuterVolumeSpecName: "dshm") pod "1b87e5da-2a88-4139-ab98-36e6a1279307" (UID: "1b87e5da-2a88-4139-ab98-36e6a1279307"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:46:31.152117 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.152009    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b87e5da-2a88-4139-ab98-36e6a1279307-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b87e5da-2a88-4139-ab98-36e6a1279307" (UID: "1b87e5da-2a88-4139-ab98-36e6a1279307"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:46:31.152117 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.152058    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b87e5da-2a88-4139-ab98-36e6a1279307-kube-api-access-g24wr" (OuterVolumeSpecName: "kube-api-access-g24wr") pod "1b87e5da-2a88-4139-ab98-36e6a1279307" (UID: "1b87e5da-2a88-4139-ab98-36e6a1279307"). InnerVolumeSpecName "kube-api-access-g24wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:46:31.224065 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.224027    2583 generic.go:356] "Generic (PLEG): container finished" podID="1b87e5da-2a88-4139-ab98-36e6a1279307" containerID="3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4" exitCode=0
Sep 06 10:46:31.224215 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.224070    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"
Sep 06 10:46:31.224215 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.224078    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57" event={"ID":"1b87e5da-2a88-4139-ab98-36e6a1279307","Type":"ContainerDied","Data":"3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4"}
Sep 06 10:46:31.224215 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.224119    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57" event={"ID":"1b87e5da-2a88-4139-ab98-36e6a1279307","Type":"ContainerDied","Data":"63263f2b66f2b8dff3798defcf424b35cce53f03a1f839ccfd88c2dc0e757b17"}
Sep 06 10:46:31.224215 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.224144    2583 scope.go:117] "RemoveContainer" containerID="3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4"
Sep 06 10:46:31.233149 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.233132    2583 scope.go:117] "RemoveContainer" containerID="3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4"
Sep 06 10:46:31.233494 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:46:31.233473    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4\": container with ID starting with 3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4 not found: ID does not exist" containerID="3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4"
Sep 06 10:46:31.233543 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.233503    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4"} err="failed to get container status \"3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4\": rpc error: code = NotFound desc = could not find container \"3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4\": container with ID starting with 3eb675c4abe0124067d2d1e037d6bc42f5eeaf686db33f4b7b495c9da06291f4 not found: ID does not exist"
Sep 06 10:46:31.244170 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.244145    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"]
Sep 06 10:46:31.247790 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.247770    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-682ea0/rate-test-model-682ea0-kserve-5f96d45459-rvp57"]
Sep 06 10:46:31.250492 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.250464    2583 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-home\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:46:31.250492 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.250494    2583 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-dshm\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:46:31.250653 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.250504    2583 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-model-cache\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:46:31.250653 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.250512    2583 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b87e5da-2a88-4139-ab98-36e6a1279307-tmp-dir\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:46:31.250653 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.250520    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g24wr\" (UniqueName: \"kubernetes.io/projected/1b87e5da-2a88-4139-ab98-36e6a1279307-kube-api-access-g24wr\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:46:31.250653 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.250529    2583 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b87e5da-2a88-4139-ab98-36e6a1279307-tls-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:46:31.485775 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:31.481883    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:46:33.239894 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:33.239862    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b87e5da-2a88-4139-ab98-36e6a1279307" path="/var/lib/kubelet/pods/1b87e5da-2a88-4139-ab98-36e6a1279307/volumes"
Sep 06 10:46:34.191785 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:34.191752    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:46:50.993458 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:50.993415    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"]
Sep 06 10:46:50.993854 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:50.993802    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b87e5da-2a88-4139-ab98-36e6a1279307" containerName="main"
Sep 06 10:46:50.993854 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:50.993815    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b87e5da-2a88-4139-ab98-36e6a1279307" containerName="main"
Sep 06 10:46:50.993928 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:50.993897    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b87e5da-2a88-4139-ab98-36e6a1279307" containerName="main"
Sep 06 10:46:50.996881 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:50.996865    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:46:50.999135 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:50.999112    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5be549\""
Sep 06 10:46:51.007752 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:51.007725    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"]
Sep 06 10:46:51.133051 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:51.133014    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8lrb\" (UniqueName: \"kubernetes.io/projected/a92e30da-d257-4de5-a471-b524683d616b-kube-api-access-q8lrb\") pod \"payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2\" (UID: \"a92e30da-d257-4de5-a471-b524683d616b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:46:51.133217 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:51.133128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e30da-d257-4de5-a471-b524683d616b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2\" (UID: \"a92e30da-d257-4de5-a471-b524683d616b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:46:51.234159 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:51.234105    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e30da-d257-4de5-a471-b524683d616b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2\" (UID: \"a92e30da-d257-4de5-a471-b524683d616b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:46:51.234386 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:51.234233    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8lrb\" (UniqueName: \"kubernetes.io/projected/a92e30da-d257-4de5-a471-b524683d616b-kube-api-access-q8lrb\") pod \"payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2\" (UID: \"a92e30da-d257-4de5-a471-b524683d616b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:46:51.235225 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:51.235202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e30da-d257-4de5-a471-b524683d616b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2\" (UID: \"a92e30da-d257-4de5-a471-b524683d616b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:46:51.242285 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:51.242255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8lrb\" (UniqueName: \"kubernetes.io/projected/a92e30da-d257-4de5-a471-b524683d616b-kube-api-access-q8lrb\") pod \"payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2\" (UID: \"a92e30da-d257-4de5-a471-b524683d616b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:46:51.307743 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:51.307663    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:46:51.581506 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:51.581466    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"]
Sep 06 10:46:51.584422 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:46:51.584389    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92e30da_d257_4de5_a471_b524683d616b.slice/crio-99684839cf615a69da527a0b5cd95bcbbc0ee8c06f140871c6058b55d31a2675 WatchSource:0}: Error finding container 99684839cf615a69da527a0b5cd95bcbbc0ee8c06f140871c6058b55d31a2675: Status 404 returned error can't find the container with id 99684839cf615a69da527a0b5cd95bcbbc0ee8c06f140871c6058b55d31a2675
Sep 06 10:46:52.333017 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:52.332983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2" event={"ID":"a92e30da-d257-4de5-a471-b524683d616b","Type":"ContainerStarted","Data":"22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff"}
Sep 06 10:46:52.333514 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:52.333070    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:46:52.333514 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:52.333091    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2" event={"ID":"a92e30da-d257-4de5-a471-b524683d616b","Type":"ContainerStarted","Data":"99684839cf615a69da527a0b5cd95bcbbc0ee8c06f140871c6058b55d31a2675"}
Sep 06 10:46:52.349121 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:52.349075    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2" podStartSLOduration=2.349061311 podStartE2EDuration="2.349061311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:52.348007353 +0000 UTC m=+1503.620648881" watchObservedRunningTime="2026-09-06 10:46:52.349061311 +0000 UTC m=+1503.621702867"
Sep 06 10:46:57.071304 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:46:57.071266    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:47:03.339890 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:47:03.339844    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:47:36.475458 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:47:36.475426    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:47:52.367908 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:47:52.367871    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:48:04.841302 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.841253    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"]
Sep 06 10:48:04.844892 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.844868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.846950 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.846929    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-622573-serving-cert\""
Sep 06 10:48:04.856472 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.856444    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"]
Sep 06 10:48:04.886124 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.886090    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72wn\" (UniqueName: \"kubernetes.io/projected/cbd48ef9-2216-4200-81cd-1672053a3348-kube-api-access-n72wn\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.886289 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.886163    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd48ef9-2216-4200-81cd-1672053a3348-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.886289 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.886200    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.886425 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.886285    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.987354 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.987290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.987525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.987387    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.987525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.987440    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72wn\" (UniqueName: \"kubernetes.io/projected/cbd48ef9-2216-4200-81cd-1672053a3348-kube-api-access-n72wn\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.987525 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.987517    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd48ef9-2216-4200-81cd-1672053a3348-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.987643 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:48:04.987524    2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-serving-cert: secret "maas-api-e2e-ipp-cleanup-622573-serving-cert" not found
Sep 06 10:48:04.987643 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:48:04.987585    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-maas-api-tls podName:cbd48ef9-2216-4200-81cd-1672053a3348 nodeName:}" failed. No retries permitted until 2026-09-06 10:48:05.487567991 +0000 UTC m=+1576.760209471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn" (UID: "cbd48ef9-2216-4200-81cd-1672053a3348") : secret "maas-api-e2e-ipp-cleanup-622573-serving-cert" not found
Sep 06 10:48:04.988122 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.988090    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd48ef9-2216-4200-81cd-1672053a3348-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.989783 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.989761    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:04.996155 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:04.996135    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72wn\" (UniqueName: \"kubernetes.io/projected/cbd48ef9-2216-4200-81cd-1672053a3348-kube-api-access-n72wn\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:05.466859 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.466823    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"]
Sep 06 10:48:05.470535 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.470511    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:05.472893 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.472816    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-622573-dockercfg-5lhs6\""
Sep 06 10:48:05.472893 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.472841    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-622573\""
Sep 06 10:48:05.479394 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.479369    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"]
Sep 06 10:48:05.492519 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.492490    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ad6e3-097b-4290-aabc-d5c59e905873-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb\" (UID: \"e13ad6e3-097b-4290-aabc-d5c59e905873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:05.492630 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.492542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjxwf\" (UniqueName: \"kubernetes.io/projected/e13ad6e3-097b-4290-aabc-d5c59e905873-kube-api-access-mjxwf\") pod \"payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb\" (UID: \"e13ad6e3-097b-4290-aabc-d5c59e905873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:05.492630 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.492592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:05.494979 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.494954    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:05.506956 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.506935    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"]
Sep 06 10:48:05.510255 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.510239    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:05.520551 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.520523    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"]
Sep 06 10:48:05.593932 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.593884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ad6e3-097b-4290-aabc-d5c59e905873-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb\" (UID: \"e13ad6e3-097b-4290-aabc-d5c59e905873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:05.594192 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.594128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjxwf\" (UniqueName: \"kubernetes.io/projected/e13ad6e3-097b-4290-aabc-d5c59e905873-kube-api-access-mjxwf\") pod \"payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb\" (UID: \"e13ad6e3-097b-4290-aabc-d5c59e905873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:05.594310 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.594247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5j8v\" (UniqueName: \"kubernetes.io/projected/fb011410-f8dd-4581-a13a-34ba41a3d8d5-kube-api-access-g5j8v\") pod \"payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87\" (UID: \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:05.594310 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.594293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb011410-f8dd-4581-a13a-34ba41a3d8d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87\" (UID: \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:05.595083 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.595056    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ad6e3-097b-4290-aabc-d5c59e905873-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb\" (UID: \"e13ad6e3-097b-4290-aabc-d5c59e905873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:05.604009 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.603986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjxwf\" (UniqueName: \"kubernetes.io/projected/e13ad6e3-097b-4290-aabc-d5c59e905873-kube-api-access-mjxwf\") pod \"payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb\" (UID: \"e13ad6e3-097b-4290-aabc-d5c59e905873\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:05.695495 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.695460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5j8v\" (UniqueName: \"kubernetes.io/projected/fb011410-f8dd-4581-a13a-34ba41a3d8d5-kube-api-access-g5j8v\") pod \"payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87\" (UID: \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:05.695495 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.695497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb011410-f8dd-4581-a13a-34ba41a3d8d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87\" (UID: \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:05.696141 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.696124    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb011410-f8dd-4581-a13a-34ba41a3d8d5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87\" (UID: \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:05.705638 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.705612    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5j8v\" (UniqueName: \"kubernetes.io/projected/fb011410-f8dd-4581-a13a-34ba41a3d8d5-kube-api-access-g5j8v\") pod \"payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87\" (UID: \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:05.756465 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.756370    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:05.781800 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.781765    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:05.825985 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:05.825946    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:06.028066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.028041    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"]
Sep 06 10:48:06.030658 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:48:06.030627    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd48ef9_2216_4200_81cd_1672053a3348.slice/crio-a2b6cd14393158bba53ab097ab530d0e5665f9d5f0c850c3e7660e46d16575f7 WatchSource:0}: Error finding container a2b6cd14393158bba53ab097ab530d0e5665f9d5f0c850c3e7660e46d16575f7: Status 404 returned error can't find the container with id a2b6cd14393158bba53ab097ab530d0e5665f9d5f0c850c3e7660e46d16575f7
Sep 06 10:48:06.064499 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.064476    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"]
Sep 06 10:48:06.066276 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:48:06.066248    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13ad6e3_097b_4290_aabc_d5c59e905873.slice/crio-128e5d84bf6dfcc8e52e45729605ac1bd25a61f1a0556b82e8f7385328a4b69e WatchSource:0}: Error finding container 128e5d84bf6dfcc8e52e45729605ac1bd25a61f1a0556b82e8f7385328a4b69e: Status 404 returned error can't find the container with id 128e5d84bf6dfcc8e52e45729605ac1bd25a61f1a0556b82e8f7385328a4b69e
Sep 06 10:48:06.113193 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.113163    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"]
Sep 06 10:48:06.116134 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:48:06.116111    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb011410_f8dd_4581_a13a_34ba41a3d8d5.slice/crio-d7cf3527dc4081cfa2574555ca11d03cb6ff466e40c9e968ec5db2e427ce3a1c WatchSource:0}: Error finding container d7cf3527dc4081cfa2574555ca11d03cb6ff466e40c9e968ec5db2e427ce3a1c: Status 404 returned error can't find the container with id d7cf3527dc4081cfa2574555ca11d03cb6ff466e40c9e968ec5db2e427ce3a1c
Sep 06 10:48:06.692851 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.692748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb" event={"ID":"e13ad6e3-097b-4290-aabc-d5c59e905873","Type":"ContainerStarted","Data":"bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3"}
Sep 06 10:48:06.692851 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.692817    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb" event={"ID":"e13ad6e3-097b-4290-aabc-d5c59e905873","Type":"ContainerStarted","Data":"128e5d84bf6dfcc8e52e45729605ac1bd25a61f1a0556b82e8f7385328a4b69e"}
Sep 06 10:48:06.693088 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.692854    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:06.694325 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.694297    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87" event={"ID":"fb011410-f8dd-4581-a13a-34ba41a3d8d5","Type":"ContainerStarted","Data":"7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd"}
Sep 06 10:48:06.694484 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.694355    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87" event={"ID":"fb011410-f8dd-4581-a13a-34ba41a3d8d5","Type":"ContainerStarted","Data":"d7cf3527dc4081cfa2574555ca11d03cb6ff466e40c9e968ec5db2e427ce3a1c"}
Sep 06 10:48:06.694484 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.694419    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:06.695933 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.695912    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn" event={"ID":"cbd48ef9-2216-4200-81cd-1672053a3348","Type":"ContainerStarted","Data":"833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2"}
Sep 06 10:48:06.696045 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.695946    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn" event={"ID":"cbd48ef9-2216-4200-81cd-1672053a3348","Type":"ContainerStarted","Data":"a2b6cd14393158bba53ab097ab530d0e5665f9d5f0c850c3e7660e46d16575f7"}
Sep 06 10:48:06.696045 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.695972    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:06.709442 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.709395    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb" podStartSLOduration=1.7093752229999999 podStartE2EDuration="1.709375223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:48:06.708822955 +0000 UTC m=+1577.981464459" watchObservedRunningTime="2026-09-06 10:48:06.709375223 +0000 UTC m=+1577.982016732"
Sep 06 10:48:06.726012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.725964    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87" podStartSLOduration=1.7259497929999998 podStartE2EDuration="1.725949793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:48:06.724432088 +0000 UTC m=+1577.997073591" watchObservedRunningTime="2026-09-06 10:48:06.725949793 +0000 UTC m=+1577.998591331"
Sep 06 10:48:06.743725 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:06.743673    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn" podStartSLOduration=2.332280302 podStartE2EDuration="2.743656891s" totalImagesPullingTime="411.376589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:48:06.742767629 +0000 UTC m=+1578.015409148" watchObservedRunningTime="2026-09-06 10:48:06.743656891 +0000 UTC m=+1578.016298392"
Sep 06 10:48:10.283614 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:10.283579    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:48:12.707633 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:12.707590    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:17.703552 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:17.703520    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:17.704167 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:17.704145    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:35.334732 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.334692    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"]
Sep 06 10:48:35.335366 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.334956    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn" podUID="cbd48ef9-2216-4200-81cd-1672053a3348" containerName="maas-api" containerID="cri-o://833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2" gracePeriod=30
Sep 06 10:48:35.384822 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.384611    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"]
Sep 06 10:48:35.385003 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.384868    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87" podUID="fb011410-f8dd-4581-a13a-34ba41a3d8d5" containerName="payload-processing" containerID="cri-o://7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd" gracePeriod=30
Sep 06 10:48:35.393469 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.393361    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"]
Sep 06 10:48:35.393637 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.393607    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb" podUID="e13ad6e3-097b-4290-aabc-d5c59e905873" containerName="payload-pre-processing" containerID="cri-o://bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3" gracePeriod=30
Sep 06 10:48:35.676357 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.676306    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:35.695129 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.695051    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:35.706737 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.706714    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:35.769734 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.769701    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-maas-api-tls\") pod \"cbd48ef9-2216-4200-81cd-1672053a3348\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") "
Sep 06 10:48:35.769920 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.769744    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n72wn\" (UniqueName: \"kubernetes.io/projected/cbd48ef9-2216-4200-81cd-1672053a3348-kube-api-access-n72wn\") pod \"cbd48ef9-2216-4200-81cd-1672053a3348\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") "
Sep 06 10:48:35.769920 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.769794    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd48ef9-2216-4200-81cd-1672053a3348-openshift-service-ca\") pod \"cbd48ef9-2216-4200-81cd-1672053a3348\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") "
Sep 06 10:48:35.769920 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.769826    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-metrics-certs\") pod \"cbd48ef9-2216-4200-81cd-1672053a3348\" (UID: \"cbd48ef9-2216-4200-81cd-1672053a3348\") "
Sep 06 10:48:35.770171 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.770144    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd48ef9-2216-4200-81cd-1672053a3348-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cbd48ef9-2216-4200-81cd-1672053a3348" (UID: "cbd48ef9-2216-4200-81cd-1672053a3348"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:48:35.771854 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.771827    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cbd48ef9-2216-4200-81cd-1672053a3348" (UID: "cbd48ef9-2216-4200-81cd-1672053a3348"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:48:35.771954 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.771850    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cbd48ef9-2216-4200-81cd-1672053a3348" (UID: "cbd48ef9-2216-4200-81cd-1672053a3348"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:48:35.771954 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.771882    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd48ef9-2216-4200-81cd-1672053a3348-kube-api-access-n72wn" (OuterVolumeSpecName: "kube-api-access-n72wn") pod "cbd48ef9-2216-4200-81cd-1672053a3348" (UID: "cbd48ef9-2216-4200-81cd-1672053a3348"). InnerVolumeSpecName "kube-api-access-n72wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:48:35.838144 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.838110    2583 generic.go:356] "Generic (PLEG): container finished" podID="fb011410-f8dd-4581-a13a-34ba41a3d8d5" containerID="7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd" exitCode=0
Sep 06 10:48:35.838349 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.838171    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"
Sep 06 10:48:35.838349 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.838193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87" event={"ID":"fb011410-f8dd-4581-a13a-34ba41a3d8d5","Type":"ContainerDied","Data":"7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd"}
Sep 06 10:48:35.838349 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.838258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87" event={"ID":"fb011410-f8dd-4581-a13a-34ba41a3d8d5","Type":"ContainerDied","Data":"d7cf3527dc4081cfa2574555ca11d03cb6ff466e40c9e968ec5db2e427ce3a1c"}
Sep 06 10:48:35.838349 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.838289    2583 scope.go:117] "RemoveContainer" containerID="7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd"
Sep 06 10:48:35.839601 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.839583    2583 generic.go:356] "Generic (PLEG): container finished" podID="cbd48ef9-2216-4200-81cd-1672053a3348" containerID="833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2" exitCode=0
Sep 06 10:48:35.839700 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.839635    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"
Sep 06 10:48:35.839700 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.839682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn" event={"ID":"cbd48ef9-2216-4200-81cd-1672053a3348","Type":"ContainerDied","Data":"833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2"}
Sep 06 10:48:35.839814 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.839736    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn" event={"ID":"cbd48ef9-2216-4200-81cd-1672053a3348","Type":"ContainerDied","Data":"a2b6cd14393158bba53ab097ab530d0e5665f9d5f0c850c3e7660e46d16575f7"}
Sep 06 10:48:35.840947 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.840929    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"
Sep 06 10:48:35.841066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.840927    2583 generic.go:356] "Generic (PLEG): container finished" podID="e13ad6e3-097b-4290-aabc-d5c59e905873" containerID="bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3" exitCode=0
Sep 06 10:48:35.841066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.840929    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb" event={"ID":"e13ad6e3-097b-4290-aabc-d5c59e905873","Type":"ContainerDied","Data":"bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3"}
Sep 06 10:48:35.841160 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.841087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb" event={"ID":"e13ad6e3-097b-4290-aabc-d5c59e905873","Type":"ContainerDied","Data":"128e5d84bf6dfcc8e52e45729605ac1bd25a61f1a0556b82e8f7385328a4b69e"}
Sep 06 10:48:35.849209 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.849179    2583 scope.go:117] "RemoveContainer" containerID="7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd"
Sep 06 10:48:35.849645 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:48:35.849612    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd\": container with ID starting with 7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd not found: ID does not exist" containerID="7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd"
Sep 06 10:48:35.849741 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.849658    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd"} err="failed to get container status \"7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd\": rpc error: code = NotFound desc = could not find container \"7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd\": container with ID starting with 7f50118eaeb8869f6c4e9afb46346b1b31d61a992410d13027ee704e9f0216dd not found: ID does not exist"
Sep 06 10:48:35.849741 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.849686    2583 scope.go:117] "RemoveContainer" containerID="833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2"
Sep 06 10:48:35.859229 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.859208    2583 scope.go:117] "RemoveContainer" containerID="833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2"
Sep 06 10:48:35.859519 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:48:35.859501    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2\": container with ID starting with 833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2 not found: ID does not exist" containerID="833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2"
Sep 06 10:48:35.859574 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.859525    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2"} err="failed to get container status \"833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2\": rpc error: code = NotFound desc = could not find container \"833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2\": container with ID starting with 833cd9c7592b58acfd29f0250627c1221d50ca63137812cf13c9375a92eb74b2 not found: ID does not exist"
Sep 06 10:48:35.859574 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.859540    2583 scope.go:117] "RemoveContainer" containerID="bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3"
Sep 06 10:48:35.865244 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.865224    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"]
Sep 06 10:48:35.868258 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.868242    2583 scope.go:117] "RemoveContainer" containerID="bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3"
Sep 06 10:48:35.868553 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:48:35.868534    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3\": container with ID starting with bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3 not found: ID does not exist" containerID="bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3"
Sep 06 10:48:35.868626 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.868564    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3"} err="failed to get container status \"bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3\": rpc error: code = NotFound desc = could not find container \"bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3\": container with ID starting with bf652908a5c0851bfd1e239f2e5e9ecd3e8376038a70ef9068a1b66db859f4e3 not found: ID does not exist"
Sep 06 10:48:35.868947 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.868929    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-622573-7ffd47c8b5-tvqfn"]
Sep 06 10:48:35.870821 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.870805    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb011410-f8dd-4581-a13a-34ba41a3d8d5-plugins-config-volume\") pod \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\" (UID: \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\") "
Sep 06 10:48:35.870908 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.870896    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjxwf\" (UniqueName: \"kubernetes.io/projected/e13ad6e3-097b-4290-aabc-d5c59e905873-kube-api-access-mjxwf\") pod \"e13ad6e3-097b-4290-aabc-d5c59e905873\" (UID: \"e13ad6e3-097b-4290-aabc-d5c59e905873\") "
Sep 06 10:48:35.870973 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.870917    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5j8v\" (UniqueName: \"kubernetes.io/projected/fb011410-f8dd-4581-a13a-34ba41a3d8d5-kube-api-access-g5j8v\") pod \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\" (UID: \"fb011410-f8dd-4581-a13a-34ba41a3d8d5\") "
Sep 06 10:48:35.870973 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.870946    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ad6e3-097b-4290-aabc-d5c59e905873-plugins-config-volume\") pod \"e13ad6e3-097b-4290-aabc-d5c59e905873\" (UID: \"e13ad6e3-097b-4290-aabc-d5c59e905873\") "
Sep 06 10:48:35.871157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.871139    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:48:35.871221 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.871163    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n72wn\" (UniqueName: \"kubernetes.io/projected/cbd48ef9-2216-4200-81cd-1672053a3348-kube-api-access-n72wn\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:48:35.871221 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.871178    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbd48ef9-2216-4200-81cd-1672053a3348-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:48:35.871221 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.871191    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbd48ef9-2216-4200-81cd-1672053a3348-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:48:35.871369 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.871252    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb011410-f8dd-4581-a13a-34ba41a3d8d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb011410-f8dd-4581-a13a-34ba41a3d8d5" (UID: "fb011410-f8dd-4581-a13a-34ba41a3d8d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:48:35.871369 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.871314    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13ad6e3-097b-4290-aabc-d5c59e905873-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e13ad6e3-097b-4290-aabc-d5c59e905873" (UID: "e13ad6e3-097b-4290-aabc-d5c59e905873"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:48:35.872843 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.872826    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb011410-f8dd-4581-a13a-34ba41a3d8d5-kube-api-access-g5j8v" (OuterVolumeSpecName: "kube-api-access-g5j8v") pod "fb011410-f8dd-4581-a13a-34ba41a3d8d5" (UID: "fb011410-f8dd-4581-a13a-34ba41a3d8d5"). InnerVolumeSpecName "kube-api-access-g5j8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:48:35.872891 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.872872    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13ad6e3-097b-4290-aabc-d5c59e905873-kube-api-access-mjxwf" (OuterVolumeSpecName: "kube-api-access-mjxwf") pod "e13ad6e3-097b-4290-aabc-d5c59e905873" (UID: "e13ad6e3-097b-4290-aabc-d5c59e905873"). InnerVolumeSpecName "kube-api-access-mjxwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:48:35.971782 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.971747    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjxwf\" (UniqueName: \"kubernetes.io/projected/e13ad6e3-097b-4290-aabc-d5c59e905873-kube-api-access-mjxwf\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:48:35.971782 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.971776    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5j8v\" (UniqueName: \"kubernetes.io/projected/fb011410-f8dd-4581-a13a-34ba41a3d8d5-kube-api-access-g5j8v\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:48:35.971782 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.971785    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e13ad6e3-097b-4290-aabc-d5c59e905873-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:48:35.972009 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:35.971794    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb011410-f8dd-4581-a13a-34ba41a3d8d5-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:48:36.160209 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:36.160177    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"]
Sep 06 10:48:36.163742 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:36.163717    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-622573-f4c9f9989-hzb87"]
Sep 06 10:48:36.173320 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:36.173295    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"]
Sep 06 10:48:36.176885 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:36.176863    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-622573-6c98469fcd-d6slb"]
Sep 06 10:48:37.240584 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:37.240543    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbd48ef9-2216-4200-81cd-1672053a3348" path="/var/lib/kubelet/pods/cbd48ef9-2216-4200-81cd-1672053a3348/volumes"
Sep 06 10:48:37.240959 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:37.240918    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e13ad6e3-097b-4290-aabc-d5c59e905873" path="/var/lib/kubelet/pods/e13ad6e3-097b-4290-aabc-d5c59e905873/volumes"
Sep 06 10:48:37.241238 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:37.241224    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb011410-f8dd-4581-a13a-34ba41a3d8d5" path="/var/lib/kubelet/pods/fb011410-f8dd-4581-a13a-34ba41a3d8d5/volumes"
Sep 06 10:48:41.970654 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:48:41.970564    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:49:01.166899 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.166859    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:49:01.306216 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.306177    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"]
Sep 06 10:49:01.306500 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.306469    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4" podUID="35b54c23-2c58-4352-8eb0-e3219d5d82a4" containerName="maas-api" containerID="cri-o://561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16" gracePeriod=30
Sep 06 10:49:01.368005 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.367970    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"]
Sep 06 10:49:01.368254 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.368227    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn" podUID="d8f5d411-1de5-4ade-986f-1d59dc69bd5d" containerName="payload-pre-processing" containerID="cri-o://00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc" gracePeriod=30
Sep 06 10:49:01.622359 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.622312    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:49:01.649804 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.649771    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:49:01.685619 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.685589    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b54c23-2c58-4352-8eb0-e3219d5d82a4-openshift-service-ca\") pod \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") "
Sep 06 10:49:01.685811 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.685634    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-metrics-certs\") pod \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") "
Sep 06 10:49:01.685811 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.685674    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-maas-api-tls\") pod \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") "
Sep 06 10:49:01.685811 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.685711    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-plugins-config-volume\") pod \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\" (UID: \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\") "
Sep 06 10:49:01.685811 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.685749    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7fj6\" (UniqueName: \"kubernetes.io/projected/35b54c23-2c58-4352-8eb0-e3219d5d82a4-kube-api-access-k7fj6\") pod \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\" (UID: \"35b54c23-2c58-4352-8eb0-e3219d5d82a4\") "
Sep 06 10:49:01.685811 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.685781    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtxz2\" (UniqueName: \"kubernetes.io/projected/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-kube-api-access-vtxz2\") pod \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\" (UID: \"d8f5d411-1de5-4ade-986f-1d59dc69bd5d\") "
Sep 06 10:49:01.686115 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.686083    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35b54c23-2c58-4352-8eb0-e3219d5d82a4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35b54c23-2c58-4352-8eb0-e3219d5d82a4" (UID: "35b54c23-2c58-4352-8eb0-e3219d5d82a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:49:01.686490 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.686439    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8f5d411-1de5-4ade-986f-1d59dc69bd5d" (UID: "d8f5d411-1de5-4ade-986f-1d59dc69bd5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:49:01.688121 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.688091    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35b54c23-2c58-4352-8eb0-e3219d5d82a4" (UID: "35b54c23-2c58-4352-8eb0-e3219d5d82a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:49:01.688269 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.688241    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "35b54c23-2c58-4352-8eb0-e3219d5d82a4" (UID: "35b54c23-2c58-4352-8eb0-e3219d5d82a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:49:01.688742 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.688718    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35b54c23-2c58-4352-8eb0-e3219d5d82a4-kube-api-access-k7fj6" (OuterVolumeSpecName: "kube-api-access-k7fj6") pod "35b54c23-2c58-4352-8eb0-e3219d5d82a4" (UID: "35b54c23-2c58-4352-8eb0-e3219d5d82a4"). InnerVolumeSpecName "kube-api-access-k7fj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:49:01.688821 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.688753    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-kube-api-access-vtxz2" (OuterVolumeSpecName: "kube-api-access-vtxz2") pod "d8f5d411-1de5-4ade-986f-1d59dc69bd5d" (UID: "d8f5d411-1de5-4ade-986f-1d59dc69bd5d"). InnerVolumeSpecName "kube-api-access-vtxz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:49:01.787391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.787289    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35b54c23-2c58-4352-8eb0-e3219d5d82a4-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:49:01.787391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.787321    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:49:01.787391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.787330    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35b54c23-2c58-4352-8eb0-e3219d5d82a4-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:49:01.787391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.787351    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:49:01.787391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.787359    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7fj6\" (UniqueName: \"kubernetes.io/projected/35b54c23-2c58-4352-8eb0-e3219d5d82a4-kube-api-access-k7fj6\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:49:01.787391 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.787368    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtxz2\" (UniqueName: \"kubernetes.io/projected/d8f5d411-1de5-4ade-986f-1d59dc69bd5d-kube-api-access-vtxz2\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:49:01.968076 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.968042    2583 generic.go:356] "Generic (PLEG): container finished" podID="35b54c23-2c58-4352-8eb0-e3219d5d82a4" containerID="561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16" exitCode=0
Sep 06 10:49:01.968261 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.968093    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"
Sep 06 10:49:01.968261 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.968127    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4" event={"ID":"35b54c23-2c58-4352-8eb0-e3219d5d82a4","Type":"ContainerDied","Data":"561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16"}
Sep 06 10:49:01.968261 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.968185    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4" event={"ID":"35b54c23-2c58-4352-8eb0-e3219d5d82a4","Type":"ContainerDied","Data":"a57c6e192580f317f3674569116a71c219a7f77b84cc78cfccd8d9d10ae4d840"}
Sep 06 10:49:01.968261 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.968217    2583 scope.go:117] "RemoveContainer" containerID="561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16"
Sep 06 10:49:01.969747 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.969724    2583 generic.go:356] "Generic (PLEG): container finished" podID="d8f5d411-1de5-4ade-986f-1d59dc69bd5d" containerID="00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc" exitCode=0
Sep 06 10:49:01.969832 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.969770    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"
Sep 06 10:49:01.969832 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.969799    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn" event={"ID":"d8f5d411-1de5-4ade-986f-1d59dc69bd5d","Type":"ContainerDied","Data":"00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc"}
Sep 06 10:49:01.969931 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.969833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn" event={"ID":"d8f5d411-1de5-4ade-986f-1d59dc69bd5d","Type":"ContainerDied","Data":"457cb8fcaca48ee96b5523ae51e4d6f8db7ef0b589a6e9127a00f3548a0d72fb"}
Sep 06 10:49:01.977595 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.977580    2583 scope.go:117] "RemoveContainer" containerID="561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16"
Sep 06 10:49:01.977891 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:49:01.977868    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16\": container with ID starting with 561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16 not found: ID does not exist" containerID="561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16"
Sep 06 10:49:01.977936 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.977904    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16"} err="failed to get container status \"561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16\": rpc error: code = NotFound desc = could not find container \"561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16\": container with ID starting with 561c61274a1e8d9ed2f4d24bf423f5926e5a2c4d9c702c8d51174b39f64f4d16 not found: ID does not exist"
Sep 06 10:49:01.977936 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.977930    2583 scope.go:117] "RemoveContainer" containerID="00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc"
Sep 06 10:49:01.986002 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.985978    2583 scope.go:117] "RemoveContainer" containerID="00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc"
Sep 06 10:49:01.986259 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:49:01.986240    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc\": container with ID starting with 00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc not found: ID does not exist" containerID="00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc"
Sep 06 10:49:01.986304 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.986266    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc"} err="failed to get container status \"00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc\": rpc error: code = NotFound desc = could not find container \"00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc\": container with ID starting with 00dea9a7d4dbc5d6f515b555880f9ed3ead4aa3fbefae548810485354456f4bc not found: ID does not exist"
Sep 06 10:49:01.992199 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.992178    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"]
Sep 06 10:49:01.996879 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:01.996858    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7d2fed-79b9c88697-4mcl4"]
Sep 06 10:49:02.009905 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:02.009874    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"]
Sep 06 10:49:02.012991 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:02.012967    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7d2fed-79bb5b5558-cl4rn"]
Sep 06 10:49:03.240329 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:03.240294    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35b54c23-2c58-4352-8eb0-e3219d5d82a4" path="/var/lib/kubelet/pods/35b54c23-2c58-4352-8eb0-e3219d5d82a4/volumes"
Sep 06 10:49:03.240719 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:03.240703    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8f5d411-1de5-4ade-986f-1d59dc69bd5d" path="/var/lib/kubelet/pods/d8f5d411-1de5-4ade-986f-1d59dc69bd5d/volumes"
Sep 06 10:49:10.665885 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:49:10.665847    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:50:00.857655 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:00.857617    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"]
Sep 06 10:50:00.858186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:00.857900    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2" podUID="a92e30da-d257-4de5-a471-b524683d616b" containerName="payload-pre-processing" containerID="cri-o://22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff" gracePeriod=30
Sep 06 10:50:01.128510 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.128485    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:50:01.211128 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.211089    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8lrb\" (UniqueName: \"kubernetes.io/projected/a92e30da-d257-4de5-a471-b524683d616b-kube-api-access-q8lrb\") pod \"a92e30da-d257-4de5-a471-b524683d616b\" (UID: \"a92e30da-d257-4de5-a471-b524683d616b\") "
Sep 06 10:50:01.211128 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.211132    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e30da-d257-4de5-a471-b524683d616b-plugins-config-volume\") pod \"a92e30da-d257-4de5-a471-b524683d616b\" (UID: \"a92e30da-d257-4de5-a471-b524683d616b\") "
Sep 06 10:50:01.211555 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.211533    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92e30da-d257-4de5-a471-b524683d616b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a92e30da-d257-4de5-a471-b524683d616b" (UID: "a92e30da-d257-4de5-a471-b524683d616b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:50:01.213186 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.213156    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92e30da-d257-4de5-a471-b524683d616b-kube-api-access-q8lrb" (OuterVolumeSpecName: "kube-api-access-q8lrb") pod "a92e30da-d257-4de5-a471-b524683d616b" (UID: "a92e30da-d257-4de5-a471-b524683d616b"). InnerVolumeSpecName "kube-api-access-q8lrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:01.253665 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.253636    2583 generic.go:356] "Generic (PLEG): container finished" podID="a92e30da-d257-4de5-a471-b524683d616b" containerID="22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff" exitCode=0
Sep 06 10:50:01.253814 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.253678    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"
Sep 06 10:50:01.253814 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.253716    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2" event={"ID":"a92e30da-d257-4de5-a471-b524683d616b","Type":"ContainerDied","Data":"22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff"}
Sep 06 10:50:01.253814 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.253777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2" event={"ID":"a92e30da-d257-4de5-a471-b524683d616b","Type":"ContainerDied","Data":"99684839cf615a69da527a0b5cd95bcbbc0ee8c06f140871c6058b55d31a2675"}
Sep 06 10:50:01.253814 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.253805    2583 scope.go:117] "RemoveContainer" containerID="22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff"
Sep 06 10:50:01.262125 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.262104    2583 scope.go:117] "RemoveContainer" containerID="22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff"
Sep 06 10:50:01.262469 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:50:01.262450    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff\": container with ID starting with 22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff not found: ID does not exist" containerID="22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff"
Sep 06 10:50:01.262539 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.262479    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff"} err="failed to get container status \"22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff\": rpc error: code = NotFound desc = could not find container \"22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff\": container with ID starting with 22cf3b9eb9e8e516357f96ac3b6a3ef8eaad81c431cbf811f69282affe9222ff not found: ID does not exist"
Sep 06 10:50:01.269621 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.269599    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"]
Sep 06 10:50:01.276103 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.276082    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5be549-69c994db94-h6lc2"]
Sep 06 10:50:01.312668 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.312633    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8lrb\" (UniqueName: \"kubernetes.io/projected/a92e30da-d257-4de5-a471-b524683d616b-kube-api-access-q8lrb\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:01.312811 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:01.312744    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a92e30da-d257-4de5-a471-b524683d616b-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:03.239401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:03.239367    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a92e30da-d257-4de5-a471-b524683d616b" path="/var/lib/kubelet/pods/a92e30da-d257-4de5-a471-b524683d616b/volumes"
Sep 06 10:50:03.875149 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:03.875104    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:50:18.441840 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.441755    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx"]
Sep 06 10:50:18.444108 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.444075    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6751d7-29811525-k4xtx"]
Sep 06 10:50:18.473389 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.473356    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"]
Sep 06 10:50:18.473900 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.473617    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h" podUID="5405b19e-e7df-4afd-8cfe-f4a48479f728" containerName="payload-pre-processing" containerID="cri-o://1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd" gracePeriod=30
Sep 06 10:50:18.738649 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.738621    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:50:18.873508 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.873476    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2bb9\" (UniqueName: \"kubernetes.io/projected/5405b19e-e7df-4afd-8cfe-f4a48479f728-kube-api-access-j2bb9\") pod \"5405b19e-e7df-4afd-8cfe-f4a48479f728\" (UID: \"5405b19e-e7df-4afd-8cfe-f4a48479f728\") "
Sep 06 10:50:18.873706 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.873579    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5405b19e-e7df-4afd-8cfe-f4a48479f728-plugins-config-volume\") pod \"5405b19e-e7df-4afd-8cfe-f4a48479f728\" (UID: \"5405b19e-e7df-4afd-8cfe-f4a48479f728\") "
Sep 06 10:50:18.873964 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.873941    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5405b19e-e7df-4afd-8cfe-f4a48479f728-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5405b19e-e7df-4afd-8cfe-f4a48479f728" (UID: "5405b19e-e7df-4afd-8cfe-f4a48479f728"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:50:18.875650 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.875627    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5405b19e-e7df-4afd-8cfe-f4a48479f728-kube-api-access-j2bb9" (OuterVolumeSpecName: "kube-api-access-j2bb9") pod "5405b19e-e7df-4afd-8cfe-f4a48479f728" (UID: "5405b19e-e7df-4afd-8cfe-f4a48479f728"). InnerVolumeSpecName "kube-api-access-j2bb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:18.974452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.974360    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5405b19e-e7df-4afd-8cfe-f4a48479f728-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:18.974452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:18.974393    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2bb9\" (UniqueName: \"kubernetes.io/projected/5405b19e-e7df-4afd-8cfe-f4a48479f728-kube-api-access-j2bb9\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:19.239941 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.239859    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d033c207-9a1e-4074-87fa-0fa93f6e005f" path="/var/lib/kubelet/pods/d033c207-9a1e-4074-87fa-0fa93f6e005f/volumes"
Sep 06 10:50:19.338663 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.338627    2583 generic.go:356] "Generic (PLEG): container finished" podID="5405b19e-e7df-4afd-8cfe-f4a48479f728" containerID="1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd" exitCode=0
Sep 06 10:50:19.338830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.338676    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"
Sep 06 10:50:19.338830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.338709    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h" event={"ID":"5405b19e-e7df-4afd-8cfe-f4a48479f728","Type":"ContainerDied","Data":"1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd"}
Sep 06 10:50:19.338830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.338759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h" event={"ID":"5405b19e-e7df-4afd-8cfe-f4a48479f728","Type":"ContainerDied","Data":"915f03a9986febc51f4ad6b80cc9cb9eae4b90d5a223d7318d42cb9c2d873538"}
Sep 06 10:50:19.338830 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.338782    2583 scope.go:117] "RemoveContainer" containerID="1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd"
Sep 06 10:50:19.347404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.347385    2583 scope.go:117] "RemoveContainer" containerID="1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd"
Sep 06 10:50:19.347718 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:50:19.347694    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd\": container with ID starting with 1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd not found: ID does not exist" containerID="1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd"
Sep 06 10:50:19.347777 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.347727    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd"} err="failed to get container status \"1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd\": rpc error: code = NotFound desc = could not find container \"1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd\": container with ID starting with 1705a80d922be3447d4bb0c5f020c805740c30ceac48a41ddc7e4a812423c6dd not found: ID does not exist"
Sep 06 10:50:19.357221 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.357197    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"]
Sep 06 10:50:19.362959 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:19.362938    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6751d7-779c8878f8-v9t9h"]
Sep 06 10:50:21.240144 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:21.240098    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5405b19e-e7df-4afd-8cfe-f4a48479f728" path="/var/lib/kubelet/pods/5405b19e-e7df-4afd-8cfe-f4a48479f728/volumes"
Sep 06 10:50:21.271044 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:21.271009    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:50:36.111759 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.111722    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"]
Sep 06 10:50:36.112290 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.111961    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g" podUID="ddb5e456-c0c5-42b6-bafa-ad4662cce6bf" containerName="payload-processing" containerID="cri-o://cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a" gracePeriod=30
Sep 06 10:50:36.118814 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.118781    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"]
Sep 06 10:50:36.119086 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.119013    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl" podUID="2658480c-969b-42c4-b007-3ea9794e1dd3" containerName="payload-pre-processing" containerID="cri-o://355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037" gracePeriod=30
Sep 06 10:50:36.395626 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.395603    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:50:36.398292 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.398228    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:50:36.419785 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.419742    2583 generic.go:356] "Generic (PLEG): container finished" podID="2658480c-969b-42c4-b007-3ea9794e1dd3" containerID="355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037" exitCode=0
Sep 06 10:50:36.419785 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.419785    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"
Sep 06 10:50:36.420082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.419823    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl" event={"ID":"2658480c-969b-42c4-b007-3ea9794e1dd3","Type":"ContainerDied","Data":"355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037"}
Sep 06 10:50:36.420082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.419910    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl" event={"ID":"2658480c-969b-42c4-b007-3ea9794e1dd3","Type":"ContainerDied","Data":"9fdf1234d2d4fa773d6f5be14baf1d96293fd726704daef694ac5c08954cbde2"}
Sep 06 10:50:36.420082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.419949    2583 scope.go:117] "RemoveContainer" containerID="355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037"
Sep 06 10:50:36.422009 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.421973    2583 generic.go:356] "Generic (PLEG): container finished" podID="ddb5e456-c0c5-42b6-bafa-ad4662cce6bf" containerID="cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a" exitCode=0
Sep 06 10:50:36.422116 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.422080    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g" event={"ID":"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf","Type":"ContainerDied","Data":"cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a"}
Sep 06 10:50:36.422185 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.422128    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g" event={"ID":"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf","Type":"ContainerDied","Data":"0e042a6a2538c50e280128b30c8d6afdcfb57f70187452bc9d4507d207eb76a6"}
Sep 06 10:50:36.422248 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.422198    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"
Sep 06 10:50:36.427670 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.427639    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-plugins-config-volume\") pod \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\" (UID: \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\") "
Sep 06 10:50:36.427797 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.427699    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvw9d\" (UniqueName: \"kubernetes.io/projected/2658480c-969b-42c4-b007-3ea9794e1dd3-kube-api-access-kvw9d\") pod \"2658480c-969b-42c4-b007-3ea9794e1dd3\" (UID: \"2658480c-969b-42c4-b007-3ea9794e1dd3\") "
Sep 06 10:50:36.427864 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.427836    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2658480c-969b-42c4-b007-3ea9794e1dd3-plugins-config-volume\") pod \"2658480c-969b-42c4-b007-3ea9794e1dd3\" (UID: \"2658480c-969b-42c4-b007-3ea9794e1dd3\") "
Sep 06 10:50:36.427917 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.427864    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnwfv\" (UniqueName: \"kubernetes.io/projected/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-kube-api-access-gnwfv\") pod \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\" (UID: \"ddb5e456-c0c5-42b6-bafa-ad4662cce6bf\") "
Sep 06 10:50:36.428809 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.428781    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ddb5e456-c0c5-42b6-bafa-ad4662cce6bf" (UID: "ddb5e456-c0c5-42b6-bafa-ad4662cce6bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:50:36.429252 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.429209    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2658480c-969b-42c4-b007-3ea9794e1dd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2658480c-969b-42c4-b007-3ea9794e1dd3" (UID: "2658480c-969b-42c4-b007-3ea9794e1dd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:50:36.431280 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.431259    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-kube-api-access-gnwfv" (OuterVolumeSpecName: "kube-api-access-gnwfv") pod "ddb5e456-c0c5-42b6-bafa-ad4662cce6bf" (UID: "ddb5e456-c0c5-42b6-bafa-ad4662cce6bf"). InnerVolumeSpecName "kube-api-access-gnwfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:36.431403 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.431290    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2658480c-969b-42c4-b007-3ea9794e1dd3-kube-api-access-kvw9d" (OuterVolumeSpecName: "kube-api-access-kvw9d") pod "2658480c-969b-42c4-b007-3ea9794e1dd3" (UID: "2658480c-969b-42c4-b007-3ea9794e1dd3"). InnerVolumeSpecName "kube-api-access-kvw9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:36.432792 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.432772    2583 scope.go:117] "RemoveContainer" containerID="355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037"
Sep 06 10:50:36.433162 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:50:36.433140    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037\": container with ID starting with 355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037 not found: ID does not exist" containerID="355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037"
Sep 06 10:50:36.433261 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.433182    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037"} err="failed to get container status \"355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037\": rpc error: code = NotFound desc = could not find container \"355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037\": container with ID starting with 355d1c96d5e0653b7734a79b30a6381bc66b3c234d22affffd4d290300158037 not found: ID does not exist"
Sep 06 10:50:36.433261 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.433199    2583 scope.go:117] "RemoveContainer" containerID="cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a"
Sep 06 10:50:36.440951 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.440934    2583 scope.go:117] "RemoveContainer" containerID="cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a"
Sep 06 10:50:36.441266 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:50:36.441247    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a\": container with ID starting with cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a not found: ID does not exist" containerID="cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a"
Sep 06 10:50:36.441356 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.441276    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a"} err="failed to get container status \"cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a\": rpc error: code = NotFound desc = could not find container \"cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a\": container with ID starting with cf41b06da59ef780ace3d1ccdf9bf432498e55ed60ac9f370c5d03b23662d22a not found: ID does not exist"
Sep 06 10:50:36.528678 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.528648    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2658480c-969b-42c4-b007-3ea9794e1dd3-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:36.528678 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.528675    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnwfv\" (UniqueName: \"kubernetes.io/projected/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-kube-api-access-gnwfv\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:36.528678 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.528685    2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf-plugins-config-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:36.528902 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.528694    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvw9d\" (UniqueName: \"kubernetes.io/projected/2658480c-969b-42c4-b007-3ea9794e1dd3-kube-api-access-kvw9d\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:36.744043 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.744014    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"]
Sep 06 10:50:36.749896 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.749872    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-682ea0-7858985658-555sl"]
Sep 06 10:50:36.759109 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.759081    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"]
Sep 06 10:50:36.762909 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:36.762888    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-682ea0-65b75d5bdf-d7k8g"]
Sep 06 10:50:37.240133 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:37.240096    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2658480c-969b-42c4-b007-3ea9794e1dd3" path="/var/lib/kubelet/pods/2658480c-969b-42c4-b007-3ea9794e1dd3/volumes"
Sep 06 10:50:37.240513 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:37.240456    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddb5e456-c0c5-42b6-bafa-ad4662cce6bf" path="/var/lib/kubelet/pods/ddb5e456-c0c5-42b6-bafa-ad4662cce6bf/volumes"
Sep 06 10:50:38.674884 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:38.674848    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:50:45.513667 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.513634    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-5r55h"]
Sep 06 10:50:45.514045 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.513872    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6ff9479bc9-5r55h" podUID="b97d71e5-238c-41ba-9b19-a5b2c2cc5273" containerName="manager" containerID="cri-o://394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72" gracePeriod=30
Sep 06 10:50:45.755775 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.755749    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:50:45.808066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.807981    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-metrics-certs\") pod \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") "
Sep 06 10:50:45.808066 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.808034    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w5c4\" (UniqueName: \"kubernetes.io/projected/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-kube-api-access-5w5c4\") pod \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") "
Sep 06 10:50:45.808287 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.808069    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-webhook-cert\") pod \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\" (UID: \"b97d71e5-238c-41ba-9b19-a5b2c2cc5273\") "
Sep 06 10:50:45.810038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.810000    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b97d71e5-238c-41ba-9b19-a5b2c2cc5273" (UID: "b97d71e5-238c-41ba-9b19-a5b2c2cc5273"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:45.810163 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.810108    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b97d71e5-238c-41ba-9b19-a5b2c2cc5273" (UID: "b97d71e5-238c-41ba-9b19-a5b2c2cc5273"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:45.810205 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.810178    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-kube-api-access-5w5c4" (OuterVolumeSpecName: "kube-api-access-5w5c4") pod "b97d71e5-238c-41ba-9b19-a5b2c2cc5273" (UID: "b97d71e5-238c-41ba-9b19-a5b2c2cc5273"). InnerVolumeSpecName "kube-api-access-5w5c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:45.909134 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.909095    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:45.909134 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.909127    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5c4\" (UniqueName: \"kubernetes.io/projected/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-kube-api-access-5w5c4\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:45.909134 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:45.909136    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b97d71e5-238c-41ba-9b19-a5b2c2cc5273-webhook-cert\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:50:46.474854 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:46.474818    2583 generic.go:356] "Generic (PLEG): container finished" podID="b97d71e5-238c-41ba-9b19-a5b2c2cc5273" containerID="394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72" exitCode=0
Sep 06 10:50:46.475031 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:46.474864    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-5r55h"
Sep 06 10:50:46.475031 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:46.474928    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-5r55h" event={"ID":"b97d71e5-238c-41ba-9b19-a5b2c2cc5273","Type":"ContainerDied","Data":"394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72"}
Sep 06 10:50:46.475031 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:46.474965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-5r55h" event={"ID":"b97d71e5-238c-41ba-9b19-a5b2c2cc5273","Type":"ContainerDied","Data":"f28aeb9936f17aa041639c5179a5838c6246f4809adb81de91ff0ce9858ed1bc"}
Sep 06 10:50:46.475031 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:46.474986    2583 scope.go:117] "RemoveContainer" containerID="394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72"
Sep 06 10:50:46.484177 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:46.484157    2583 scope.go:117] "RemoveContainer" containerID="394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72"
Sep 06 10:50:46.484478 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:50:46.484459    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72\": container with ID starting with 394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72 not found: ID does not exist" containerID="394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72"
Sep 06 10:50:46.484535 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:46.484488    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72"} err="failed to get container status \"394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72\": rpc error: code = NotFound desc = could not find container \"394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72\": container with ID starting with 394ee26b75750eeb8e848678c31d24fcdebe682476ee67b912b9652679a3ad72 not found: ID does not exist"
Sep 06 10:50:46.494273 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:46.494241    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-5r55h"]
Sep 06 10:50:46.498000 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:46.497975    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-5r55h"]
Sep 06 10:50:47.240981 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:47.240951    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b97d71e5-238c-41ba-9b19-a5b2c2cc5273" path="/var/lib/kubelet/pods/b97d71e5-238c-41ba-9b19-a5b2c2cc5273/volumes"
Sep 06 10:50:47.669670 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:50:47.669636    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:51:47.305862 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.305779    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-mrqhn"]
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306258    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2658480c-969b-42c4-b007-3ea9794e1dd3" containerName="payload-pre-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306277    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2658480c-969b-42c4-b007-3ea9794e1dd3" containerName="payload-pre-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306294    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b97d71e5-238c-41ba-9b19-a5b2c2cc5273" containerName="manager"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306302    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b97d71e5-238c-41ba-9b19-a5b2c2cc5273" containerName="manager"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306315    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8f5d411-1de5-4ade-986f-1d59dc69bd5d" containerName="payload-pre-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306324    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8f5d411-1de5-4ade-986f-1d59dc69bd5d" containerName="payload-pre-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306347    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb011410-f8dd-4581-a13a-34ba41a3d8d5" containerName="payload-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306356    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb011410-f8dd-4581-a13a-34ba41a3d8d5" containerName="payload-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306368    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a92e30da-d257-4de5-a471-b524683d616b" containerName="payload-pre-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306377    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a92e30da-d257-4de5-a471-b524683d616b" containerName="payload-pre-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306395    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddb5e456-c0c5-42b6-bafa-ad4662cce6bf" containerName="payload-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306402    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddb5e456-c0c5-42b6-bafa-ad4662cce6bf" containerName="payload-processing"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306414    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35b54c23-2c58-4352-8eb0-e3219d5d82a4" containerName="maas-api"
Sep 06 10:51:47.306411 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306421    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="35b54c23-2c58-4352-8eb0-e3219d5d82a4" containerName="maas-api"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306429    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e13ad6e3-097b-4290-aabc-d5c59e905873" containerName="payload-pre-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306438    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e13ad6e3-097b-4290-aabc-d5c59e905873" containerName="payload-pre-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306449    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5405b19e-e7df-4afd-8cfe-f4a48479f728" containerName="payload-pre-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306457    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5405b19e-e7df-4afd-8cfe-f4a48479f728" containerName="payload-pre-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306470    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbd48ef9-2216-4200-81cd-1672053a3348" containerName="maas-api"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306478    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbd48ef9-2216-4200-81cd-1672053a3348" containerName="maas-api"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306565    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a92e30da-d257-4de5-a471-b524683d616b" containerName="payload-pre-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306575    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b97d71e5-238c-41ba-9b19-a5b2c2cc5273" containerName="manager"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306585    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e13ad6e3-097b-4290-aabc-d5c59e905873" containerName="payload-pre-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306594    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbd48ef9-2216-4200-81cd-1672053a3348" containerName="maas-api"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306603    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5405b19e-e7df-4afd-8cfe-f4a48479f728" containerName="payload-pre-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306612    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="35b54c23-2c58-4352-8eb0-e3219d5d82a4" containerName="maas-api"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306623    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb011410-f8dd-4581-a13a-34ba41a3d8d5" containerName="payload-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306633    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2658480c-969b-42c4-b007-3ea9794e1dd3" containerName="payload-pre-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306641    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddb5e456-c0c5-42b6-bafa-ad4662cce6bf" containerName="payload-processing"
Sep 06 10:51:47.307098 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.306650    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8f5d411-1de5-4ade-986f-1d59dc69bd5d" containerName="payload-pre-processing"
Sep 06 10:51:47.309857 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.309836    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.312515 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.312489    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 06 10:51:47.312515 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.312505    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 06 10:51:47.312680 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.312499    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m7lb4\""
Sep 06 10:51:47.312680 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.312605    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 06 10:51:47.318665 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.318636    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-mrqhn"]
Sep 06 10:51:47.427052 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.427019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgkqp\" (UniqueName: \"kubernetes.io/projected/b28c28f9-1647-43fd-9ee6-b44e961c31f7-kube-api-access-qgkqp\") pod \"maas-controller-6ff9479bc9-mrqhn\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") " pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.427246 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.427107    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-webhook-cert\") pod \"maas-controller-6ff9479bc9-mrqhn\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") " pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.427246 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.427141    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-metrics-certs\") pod \"maas-controller-6ff9479bc9-mrqhn\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") " pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.528126 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.528090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-webhook-cert\") pod \"maas-controller-6ff9479bc9-mrqhn\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") " pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.528298 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.528153    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-metrics-certs\") pod \"maas-controller-6ff9479bc9-mrqhn\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") " pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.528298 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.528202    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgkqp\" (UniqueName: \"kubernetes.io/projected/b28c28f9-1647-43fd-9ee6-b44e961c31f7-kube-api-access-qgkqp\") pod \"maas-controller-6ff9479bc9-mrqhn\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") " pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.530708 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.530651    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-metrics-certs\") pod \"maas-controller-6ff9479bc9-mrqhn\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") " pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.531038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.530882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-webhook-cert\") pod \"maas-controller-6ff9479bc9-mrqhn\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") " pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.536012 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.535988    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgkqp\" (UniqueName: \"kubernetes.io/projected/b28c28f9-1647-43fd-9ee6-b44e961c31f7-kube-api-access-qgkqp\") pod \"maas-controller-6ff9479bc9-mrqhn\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") " pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.620641 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.620612    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:47.887928 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.887843    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-mrqhn"]
Sep 06 10:51:47.890982 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:51:47.890947    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28c28f9_1647_43fd_9ee6_b44e961c31f7.slice/crio-1da5c6466d94ead7a5beedc499c4dcdf36e074b65626b6df27326af9543bd2a9 WatchSource:0}: Error finding container 1da5c6466d94ead7a5beedc499c4dcdf36e074b65626b6df27326af9543bd2a9: Status 404 returned error can't find the container with id 1da5c6466d94ead7a5beedc499c4dcdf36e074b65626b6df27326af9543bd2a9
Sep 06 10:51:47.892252 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:47.892235    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:51:48.758000 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:48.757960    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn" event={"ID":"b28c28f9-1647-43fd-9ee6-b44e961c31f7","Type":"ContainerStarted","Data":"63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d"}
Sep 06 10:51:48.758404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:48.758019    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn" event={"ID":"b28c28f9-1647-43fd-9ee6-b44e961c31f7","Type":"ContainerStarted","Data":"1da5c6466d94ead7a5beedc499c4dcdf36e074b65626b6df27326af9543bd2a9"}
Sep 06 10:51:48.758404 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:48.758059    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:51:48.778162 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:48.778089    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn" podStartSLOduration=1.359422672 podStartE2EDuration="1.778065376s" totalImagesPullingTime="418.642704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:51:48.777615054 +0000 UTC m=+1800.050256557" watchObservedRunningTime="2026-09-06 10:51:48.778065376 +0000 UTC m=+1800.050706879"
Sep 06 10:51:49.486602 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:49.486571    2583 scope.go:117] "RemoveContainer" containerID="a2951adad88b1bad8e8ccfa0bb6026c224071ccafe7590e47c941e99b1c1cd98"
Sep 06 10:51:59.764496 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:51:59.764466    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:52:10.668808 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:10.668766    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:52:11.242280 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.242237    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-mrqhn"]
Sep 06 10:52:11.242558 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.242528    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn" podUID="b28c28f9-1647-43fd-9ee6-b44e961c31f7" containerName="manager" containerID="cri-o://63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d" gracePeriod=30
Sep 06 10:52:11.479349 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.479313    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:52:11.538546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.538454    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgkqp\" (UniqueName: \"kubernetes.io/projected/b28c28f9-1647-43fd-9ee6-b44e961c31f7-kube-api-access-qgkqp\") pod \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") "
Sep 06 10:52:11.538546 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.538534    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-metrics-certs\") pod \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") "
Sep 06 10:52:11.538734 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.538553    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-webhook-cert\") pod \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\" (UID: \"b28c28f9-1647-43fd-9ee6-b44e961c31f7\") "
Sep 06 10:52:11.540584 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.540548    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b28c28f9-1647-43fd-9ee6-b44e961c31f7" (UID: "b28c28f9-1647-43fd-9ee6-b44e961c31f7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:52:11.540730 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.540606    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b28c28f9-1647-43fd-9ee6-b44e961c31f7-kube-api-access-qgkqp" (OuterVolumeSpecName: "kube-api-access-qgkqp") pod "b28c28f9-1647-43fd-9ee6-b44e961c31f7" (UID: "b28c28f9-1647-43fd-9ee6-b44e961c31f7"). InnerVolumeSpecName "kube-api-access-qgkqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:52:11.540730 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.540629    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b28c28f9-1647-43fd-9ee6-b44e961c31f7" (UID: "b28c28f9-1647-43fd-9ee6-b44e961c31f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:52:11.639502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.639464    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:52:11.639502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.639494    2583 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b28c28f9-1647-43fd-9ee6-b44e961c31f7-webhook-cert\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:52:11.639502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.639507    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgkqp\" (UniqueName: \"kubernetes.io/projected/b28c28f9-1647-43fd-9ee6-b44e961c31f7-kube-api-access-qgkqp\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:52:11.857451 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.857419    2583 generic.go:356] "Generic (PLEG): container finished" podID="b28c28f9-1647-43fd-9ee6-b44e961c31f7" containerID="63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d" exitCode=0
Sep 06 10:52:11.857844 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.857463    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn"
Sep 06 10:52:11.857844 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.857507    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn" event={"ID":"b28c28f9-1647-43fd-9ee6-b44e961c31f7","Type":"ContainerDied","Data":"63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d"}
Sep 06 10:52:11.857844 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.857566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-mrqhn" event={"ID":"b28c28f9-1647-43fd-9ee6-b44e961c31f7","Type":"ContainerDied","Data":"1da5c6466d94ead7a5beedc499c4dcdf36e074b65626b6df27326af9543bd2a9"}
Sep 06 10:52:11.857844 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.857594    2583 scope.go:117] "RemoveContainer" containerID="63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d"
Sep 06 10:52:11.865860 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.865841    2583 scope.go:117] "RemoveContainer" containerID="63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d"
Sep 06 10:52:11.866166 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:52:11.866142    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d\": container with ID starting with 63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d not found: ID does not exist" containerID="63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d"
Sep 06 10:52:11.866249 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.866180    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d"} err="failed to get container status \"63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d\": rpc error: code = NotFound desc = could not find container \"63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d\": container with ID starting with 63917e6958923ab2fc2836d1ccb10b189152048798f5761369b96bd513d8e78d not found: ID does not exist"
Sep 06 10:52:11.878317 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.878292    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-mrqhn"]
Sep 06 10:52:11.881708 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:11.881688    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-mrqhn"]
Sep 06 10:52:12.675269 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.675237    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-hcnng"]
Sep 06 10:52:12.675587 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.675575    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b28c28f9-1647-43fd-9ee6-b44e961c31f7" containerName="manager"
Sep 06 10:52:12.675634 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.675588    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b28c28f9-1647-43fd-9ee6-b44e961c31f7" containerName="manager"
Sep 06 10:52:12.675670 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.675649    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b28c28f9-1647-43fd-9ee6-b44e961c31f7" containerName="manager"
Sep 06 10:52:12.679833 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.679812    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.682157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.682131    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 06 10:52:12.682157 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.682146    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 06 10:52:12.682306 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.682196    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-m7lb4\""
Sep 06 10:52:12.682306 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.682147    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 06 10:52:12.686223 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.686201    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-hcnng"]
Sep 06 10:52:12.749209 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.749175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtv9z\" (UniqueName: \"kubernetes.io/projected/2f2726ea-8703-407e-97f9-333fc0ceb397-kube-api-access-mtv9z\") pod \"maas-controller-6ff9479bc9-hcnng\" (UID: \"2f2726ea-8703-407e-97f9-333fc0ceb397\") " pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.749412 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.749221    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f2726ea-8703-407e-97f9-333fc0ceb397-webhook-cert\") pod \"maas-controller-6ff9479bc9-hcnng\" (UID: \"2f2726ea-8703-407e-97f9-333fc0ceb397\") " pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.749412 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.749239    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f2726ea-8703-407e-97f9-333fc0ceb397-metrics-certs\") pod \"maas-controller-6ff9479bc9-hcnng\" (UID: \"2f2726ea-8703-407e-97f9-333fc0ceb397\") " pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.850585 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.850497    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f2726ea-8703-407e-97f9-333fc0ceb397-webhook-cert\") pod \"maas-controller-6ff9479bc9-hcnng\" (UID: \"2f2726ea-8703-407e-97f9-333fc0ceb397\") " pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.850753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.850616    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f2726ea-8703-407e-97f9-333fc0ceb397-metrics-certs\") pod \"maas-controller-6ff9479bc9-hcnng\" (UID: \"2f2726ea-8703-407e-97f9-333fc0ceb397\") " pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.850753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.850728    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtv9z\" (UniqueName: \"kubernetes.io/projected/2f2726ea-8703-407e-97f9-333fc0ceb397-kube-api-access-mtv9z\") pod \"maas-controller-6ff9479bc9-hcnng\" (UID: \"2f2726ea-8703-407e-97f9-333fc0ceb397\") " pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.852980 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.852955    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f2726ea-8703-407e-97f9-333fc0ceb397-metrics-certs\") pod \"maas-controller-6ff9479bc9-hcnng\" (UID: \"2f2726ea-8703-407e-97f9-333fc0ceb397\") " pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.853114 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.852979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2f2726ea-8703-407e-97f9-333fc0ceb397-webhook-cert\") pod \"maas-controller-6ff9479bc9-hcnng\" (UID: \"2f2726ea-8703-407e-97f9-333fc0ceb397\") " pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.857995 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.857974    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtv9z\" (UniqueName: \"kubernetes.io/projected/2f2726ea-8703-407e-97f9-333fc0ceb397-kube-api-access-mtv9z\") pod \"maas-controller-6ff9479bc9-hcnng\" (UID: \"2f2726ea-8703-407e-97f9-333fc0ceb397\") " pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:12.991487 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:12.991397    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:13.241038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:13.241001    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b28c28f9-1647-43fd-9ee6-b44e961c31f7" path="/var/lib/kubelet/pods/b28c28f9-1647-43fd-9ee6-b44e961c31f7/volumes"
Sep 06 10:52:13.261668 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:13.261586    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6ff9479bc9-hcnng"]
Sep 06 10:52:13.264315 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:52:13.264288    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2726ea_8703_407e_97f9_333fc0ceb397.slice/crio-3bc6fbfc34ca97b1e49275c8541645184f1efcbb30f8b54123d5b3ed33447186 WatchSource:0}: Error finding container 3bc6fbfc34ca97b1e49275c8541645184f1efcbb30f8b54123d5b3ed33447186: Status 404 returned error can't find the container with id 3bc6fbfc34ca97b1e49275c8541645184f1efcbb30f8b54123d5b3ed33447186
Sep 06 10:52:13.869552 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:13.869513    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-hcnng" event={"ID":"2f2726ea-8703-407e-97f9-333fc0ceb397","Type":"ContainerStarted","Data":"1208fcd47c5b80275fdc75905c31506d868bd0159c6817e2ff4267558547b267"}
Sep 06 10:52:13.869950 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:13.869582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6ff9479bc9-hcnng" event={"ID":"2f2726ea-8703-407e-97f9-333fc0ceb397","Type":"ContainerStarted","Data":"3bc6fbfc34ca97b1e49275c8541645184f1efcbb30f8b54123d5b3ed33447186"}
Sep 06 10:52:13.869950 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:13.869708    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:13.885791 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:13.885740    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6ff9479bc9-hcnng" podStartSLOduration=1.5688759540000001 podStartE2EDuration="1.885723852s" totalImagesPullingTime="316.847898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:52:13.884614252 +0000 UTC m=+1825.157255768" watchObservedRunningTime="2026-09-06 10:52:13.885723852 +0000 UTC m=+1825.158365354"
Sep 06 10:52:24.877463 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:24.877430    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6ff9479bc9-hcnng"
Sep 06 10:52:36.681179 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:36.681128    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:52:44.468603 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:44.468560    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:52:52.672897 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:52:52.672859    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:53:12.270813 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:12.270733    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:53:19.167520 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:19.167481    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:53:22.441195 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:22.441154    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"]
Sep 06 10:53:22.441653 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:22.441434    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp" podUID="1dcc0480-e53e-405f-8d89-3168df03059c" containerName="manager" containerID="cri-o://3fbebc155cec47049c001f0fc08d5ac305391303f60a029052283b0b3d754a7b" gracePeriod=10
Sep 06 10:53:23.174502 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:23.174465    2583 generic.go:356] "Generic (PLEG): container finished" podID="1dcc0480-e53e-405f-8d89-3168df03059c" containerID="3fbebc155cec47049c001f0fc08d5ac305391303f60a029052283b0b3d754a7b" exitCode=0
Sep 06 10:53:23.174673 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:23.174535    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp" event={"ID":"1dcc0480-e53e-405f-8d89-3168df03059c","Type":"ContainerDied","Data":"3fbebc155cec47049c001f0fc08d5ac305391303f60a029052283b0b3d754a7b"}
Sep 06 10:53:23.287162 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:23.287138    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:53:23.358002 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:23.357968    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dcc0480-e53e-405f-8d89-3168df03059c-extensions-socket-volume\") pod \"1dcc0480-e53e-405f-8d89-3168df03059c\" (UID: \"1dcc0480-e53e-405f-8d89-3168df03059c\") "
Sep 06 10:53:23.358200 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:23.358033    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkgfh\" (UniqueName: \"kubernetes.io/projected/1dcc0480-e53e-405f-8d89-3168df03059c-kube-api-access-xkgfh\") pod \"1dcc0480-e53e-405f-8d89-3168df03059c\" (UID: \"1dcc0480-e53e-405f-8d89-3168df03059c\") "
Sep 06 10:53:23.358349 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:23.358311    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dcc0480-e53e-405f-8d89-3168df03059c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1dcc0480-e53e-405f-8d89-3168df03059c" (UID: "1dcc0480-e53e-405f-8d89-3168df03059c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:53:23.359993 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:23.359972    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dcc0480-e53e-405f-8d89-3168df03059c-kube-api-access-xkgfh" (OuterVolumeSpecName: "kube-api-access-xkgfh") pod "1dcc0480-e53e-405f-8d89-3168df03059c" (UID: "1dcc0480-e53e-405f-8d89-3168df03059c"). InnerVolumeSpecName "kube-api-access-xkgfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:53:23.459330 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:23.459242    2583 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1dcc0480-e53e-405f-8d89-3168df03059c-extensions-socket-volume\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:53:23.459330 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:23.459275    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkgfh\" (UniqueName: \"kubernetes.io/projected/1dcc0480-e53e-405f-8d89-3168df03059c-kube-api-access-xkgfh\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:53:24.180133 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:24.180102    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"
Sep 06 10:53:24.180133 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:24.180130    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp" event={"ID":"1dcc0480-e53e-405f-8d89-3168df03059c","Type":"ContainerDied","Data":"9400e6cd07fda67a1ad9a60eac9d73d64162a6a677b6455ea68d289dc2c69576"}
Sep 06 10:53:24.180440 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:24.180187    2583 scope.go:117] "RemoveContainer" containerID="3fbebc155cec47049c001f0fc08d5ac305391303f60a029052283b0b3d754a7b"
Sep 06 10:53:24.200044 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:24.200015    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"]
Sep 06 10:53:24.203728 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:24.203707    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-r4clp"]
Sep 06 10:53:25.239356 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:53:25.239311    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dcc0480-e53e-405f-8d89-3168df03059c" path="/var/lib/kubelet/pods/1dcc0480-e53e-405f-8d89-3168df03059c/volumes"
Sep 06 10:54:23.142726 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.142690    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"]
Sep 06 10:54:23.143199 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.143001    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dcc0480-e53e-405f-8d89-3168df03059c" containerName="manager"
Sep 06 10:54:23.143199 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.143011    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dcc0480-e53e-405f-8d89-3168df03059c" containerName="manager"
Sep 06 10:54:23.143199 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.143071    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dcc0480-e53e-405f-8d89-3168df03059c" containerName="manager"
Sep 06 10:54:23.145944 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.145924    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:23.148043 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.148021    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-frvzt\""
Sep 06 10:54:23.157980 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.157957    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"]
Sep 06 10:54:23.266141 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.266107    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e222e6f-6645-4bfc-82c7-36860d344b20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9q6rs\" (UID: \"0e222e6f-6645-4bfc-82c7-36860d344b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:23.266141 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.266145    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwxs\" (UniqueName: \"kubernetes.io/projected/0e222e6f-6645-4bfc-82c7-36860d344b20-kube-api-access-5wwxs\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9q6rs\" (UID: \"0e222e6f-6645-4bfc-82c7-36860d344b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:23.367094 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.367057    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e222e6f-6645-4bfc-82c7-36860d344b20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9q6rs\" (UID: \"0e222e6f-6645-4bfc-82c7-36860d344b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:23.367094 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.367098    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwxs\" (UniqueName: \"kubernetes.io/projected/0e222e6f-6645-4bfc-82c7-36860d344b20-kube-api-access-5wwxs\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9q6rs\" (UID: \"0e222e6f-6645-4bfc-82c7-36860d344b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:23.367452 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.367432    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e222e6f-6645-4bfc-82c7-36860d344b20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9q6rs\" (UID: \"0e222e6f-6645-4bfc-82c7-36860d344b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:23.375427 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.375394    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwxs\" (UniqueName: \"kubernetes.io/projected/0e222e6f-6645-4bfc-82c7-36860d344b20-kube-api-access-5wwxs\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-9q6rs\" (UID: \"0e222e6f-6645-4bfc-82c7-36860d344b20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:23.456102 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.456009    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:23.731753 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:23.731671    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"]
Sep 06 10:54:23.735637 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:54:23.735606    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e222e6f_6645_4bfc_82c7_36860d344b20.slice/crio-4e72b92bd88b2c090f960eb526cd3c8fd21dc4689272ad9b80412c041a4ca342 WatchSource:0}: Error finding container 4e72b92bd88b2c090f960eb526cd3c8fd21dc4689272ad9b80412c041a4ca342: Status 404 returned error can't find the container with id 4e72b92bd88b2c090f960eb526cd3c8fd21dc4689272ad9b80412c041a4ca342
Sep 06 10:54:24.431859 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:24.431820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs" event={"ID":"0e222e6f-6645-4bfc-82c7-36860d344b20","Type":"ContainerStarted","Data":"08959732709ae4fb1d9715fb6d66356ac1c47a457c41f1fa5db7f4ef2f9bb840"}
Sep 06 10:54:24.432238 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:24.431888    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:24.432238 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:24.431915    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs" event={"ID":"0e222e6f-6645-4bfc-82c7-36860d344b20","Type":"ContainerStarted","Data":"4e72b92bd88b2c090f960eb526cd3c8fd21dc4689272ad9b80412c041a4ca342"}
Sep 06 10:54:24.452955 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:24.452907    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs" podStartSLOduration=1.45289104 podStartE2EDuration="1.45289104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:54:24.450892083 +0000 UTC m=+1955.723533606" watchObservedRunningTime="2026-09-06 10:54:24.45289104 +0000 UTC m=+1955.725532540"
Sep 06 10:54:35.438168 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:35.438137    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-9q6rs"
Sep 06 10:54:47.073718 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:47.073634    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:54:54.872172 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:54:54.872131    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:55:14.177398 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:14.177358    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:55:22.580607 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:22.580569    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:55:47.181891 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:47.181856    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:55:50.270567 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:50.270530    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:55:56.873913 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:56.873883    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:55:57.283493 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.283414    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58fd75565c-jj787"]
Sep 06 10:55:57.295715 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.295689    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.296641 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.296603    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58fd75565c-jj787"]
Sep 06 10:55:57.376401 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.376368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7skx5\" (UniqueName: \"kubernetes.io/projected/27a75fd1-ec38-4f57-aabc-448b95e74fb6-kube-api-access-7skx5\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.376566 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.376466    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a75fd1-ec38-4f57-aabc-448b95e74fb6-metrics-certs\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.376566 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.376542    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27a75fd1-ec38-4f57-aabc-448b95e74fb6-maas-api-tls\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.376640 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.376607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a75fd1-ec38-4f57-aabc-448b95e74fb6-openshift-service-ca\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.478037 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.477998    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7skx5\" (UniqueName: \"kubernetes.io/projected/27a75fd1-ec38-4f57-aabc-448b95e74fb6-kube-api-access-7skx5\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.478197 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.478049    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a75fd1-ec38-4f57-aabc-448b95e74fb6-metrics-certs\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.478197 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.478091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27a75fd1-ec38-4f57-aabc-448b95e74fb6-maas-api-tls\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.478197 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.478143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a75fd1-ec38-4f57-aabc-448b95e74fb6-openshift-service-ca\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.478953 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.478919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27a75fd1-ec38-4f57-aabc-448b95e74fb6-openshift-service-ca\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.480626 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.480603    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27a75fd1-ec38-4f57-aabc-448b95e74fb6-maas-api-tls\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.480768 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.480747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27a75fd1-ec38-4f57-aabc-448b95e74fb6-metrics-certs\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.485184 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.485153    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7skx5\" (UniqueName: \"kubernetes.io/projected/27a75fd1-ec38-4f57-aabc-448b95e74fb6-kube-api-access-7skx5\") pod \"maas-api-58fd75565c-jj787\" (UID: \"27a75fd1-ec38-4f57-aabc-448b95e74fb6\") " pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.607078 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.607042    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:57.881179 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:57.881147    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58fd75565c-jj787"]
Sep 06 10:55:57.883540 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:55:57.883513    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a75fd1_ec38_4f57_aabc_448b95e74fb6.slice/crio-1d0e67dd9a26821dfec64a56c13e522e5152c7d3408e098cadf8c4440e7dd722 WatchSource:0}: Error finding container 1d0e67dd9a26821dfec64a56c13e522e5152c7d3408e098cadf8c4440e7dd722: Status 404 returned error can't find the container with id 1d0e67dd9a26821dfec64a56c13e522e5152c7d3408e098cadf8c4440e7dd722
Sep 06 10:55:58.853716 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:58.853682    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787" event={"ID":"27a75fd1-ec38-4f57-aabc-448b95e74fb6","Type":"ContainerStarted","Data":"e0ba6a3c031459f003bd4d1e1a7c1877582b95e8bc514298764722219e49c631"}
Sep 06 10:55:58.853925 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:58.853734    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787" event={"ID":"27a75fd1-ec38-4f57-aabc-448b95e74fb6","Type":"ContainerStarted","Data":"1d0e67dd9a26821dfec64a56c13e522e5152c7d3408e098cadf8c4440e7dd722"}
Sep 06 10:55:58.853925 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:58.853764    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:55:58.871584 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:55:58.871534    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787" podStartSLOduration=1.441387335 podStartE2EDuration="1.871517917s" totalImagesPullingTime="430.130582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:55:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:55:58.869576274 +0000 UTC m=+2050.142217788" watchObservedRunningTime="2026-09-06 10:55:58.871517917 +0000 UTC m=+2050.144159478"
Sep 06 10:56:04.863939 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:04.863911    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58fd75565c-jj787"
Sep 06 10:56:04.911717 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:04.911683    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-m767q"]
Sep 06 10:56:04.911937 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:04.911907    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q" podUID="4e5f7374-5300-400d-adc7-f47a19046797" containerName="maas-api" containerID="cri-o://311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7" gracePeriod=30
Sep 06 10:56:04.980352 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:04.980303    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:56:05.460718 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:05.460651    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q" podUID="4e5f7374-5300-400d-adc7-f47a19046797" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.31:8443/health\": dial tcp 10.132.0.31:8443: connect: connection refused"
Sep 06 10:56:06.152442 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.152413    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:56:06.259449 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.259352    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-maas-api-tls\") pod \"4e5f7374-5300-400d-adc7-f47a19046797\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") "
Sep 06 10:56:06.259449 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.259398    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-metrics-certs\") pod \"4e5f7374-5300-400d-adc7-f47a19046797\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") "
Sep 06 10:56:06.259449 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.259431    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2q62\" (UniqueName: \"kubernetes.io/projected/4e5f7374-5300-400d-adc7-f47a19046797-kube-api-access-p2q62\") pod \"4e5f7374-5300-400d-adc7-f47a19046797\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") "
Sep 06 10:56:06.259715 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.259491    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5f7374-5300-400d-adc7-f47a19046797-openshift-service-ca\") pod \"4e5f7374-5300-400d-adc7-f47a19046797\" (UID: \"4e5f7374-5300-400d-adc7-f47a19046797\") "
Sep 06 10:56:06.259901 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.259866    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5f7374-5300-400d-adc7-f47a19046797-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e5f7374-5300-400d-adc7-f47a19046797" (UID: "4e5f7374-5300-400d-adc7-f47a19046797"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:56:06.261504 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.261474    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4e5f7374-5300-400d-adc7-f47a19046797" (UID: "4e5f7374-5300-400d-adc7-f47a19046797"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:56:06.261594 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.261547    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5f7374-5300-400d-adc7-f47a19046797-kube-api-access-p2q62" (OuterVolumeSpecName: "kube-api-access-p2q62") pod "4e5f7374-5300-400d-adc7-f47a19046797" (UID: "4e5f7374-5300-400d-adc7-f47a19046797"). InnerVolumeSpecName "kube-api-access-p2q62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:56:06.261594 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.261562    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e5f7374-5300-400d-adc7-f47a19046797" (UID: "4e5f7374-5300-400d-adc7-f47a19046797"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:56:06.360192 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.360156    2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-maas-api-tls\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:56:06.360192 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.360188    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e5f7374-5300-400d-adc7-f47a19046797-metrics-certs\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:56:06.360423 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.360202    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2q62\" (UniqueName: \"kubernetes.io/projected/4e5f7374-5300-400d-adc7-f47a19046797-kube-api-access-p2q62\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:56:06.360423 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.360217    2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e5f7374-5300-400d-adc7-f47a19046797-openshift-service-ca\") on node \"ip-10-0-130-109.ec2.internal\" DevicePath \"\""
Sep 06 10:56:06.890735 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.890699    2583 generic.go:356] "Generic (PLEG): container finished" podID="4e5f7374-5300-400d-adc7-f47a19046797" containerID="311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7" exitCode=0
Sep 06 10:56:06.890906 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.890741    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q"
Sep 06 10:56:06.890906 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.890782    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q" event={"ID":"4e5f7374-5300-400d-adc7-f47a19046797","Type":"ContainerDied","Data":"311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7"}
Sep 06 10:56:06.890906 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.890832    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-679c995ddf-m767q" event={"ID":"4e5f7374-5300-400d-adc7-f47a19046797","Type":"ContainerDied","Data":"51e2954988074c12556f4bca2ece51b08d8c74724509191f09d83adb8134ccff"}
Sep 06 10:56:06.890906 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.890859    2583 scope.go:117] "RemoveContainer" containerID="311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7"
Sep 06 10:56:06.900175 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.900160    2583 scope.go:117] "RemoveContainer" containerID="311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7"
Sep 06 10:56:06.900460 ip-10-0-130-109 kubenswrapper[2583]: E0906 10:56:06.900442    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7\": container with ID starting with 311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7 not found: ID does not exist" containerID="311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7"
Sep 06 10:56:06.900539 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.900469    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7"} err="failed to get container status \"311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7\": rpc error: code = NotFound desc = could not find container \"311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7\": container with ID starting with 311416203b1291205a14d54ab72d3a2daa852a2a910d88c351512858cd0a8cc7 not found: ID does not exist"
Sep 06 10:56:06.912038 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.912013    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-m767q"]
Sep 06 10:56:06.917305 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:06.917281    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-679c995ddf-m767q"]
Sep 06 10:56:07.239988 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:07.239910    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e5f7374-5300-400d-adc7-f47a19046797" path="/var/lib/kubelet/pods/4e5f7374-5300-400d-adc7-f47a19046797/volumes"
Sep 06 10:56:11.675264 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:11.675185    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:56:20.882746 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:20.882706    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:56:28.868491 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:28.868458    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-2n59j"]
Sep 06 10:56:36.049193 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:36.049154    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ff9479bc9-hcnng_2f2726ea-8703-407e-97f9-333fc0ceb397/manager/0.log"
Sep 06 10:56:36.160312 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:36.160281    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nxqbr_474c1520-fc8c-4049-86f3-4d2331b70da1/server/0.log"
Sep 06 10:56:36.277858 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:36.277831    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8vnx9_46181b28-9d0e-4374-96a9-7df450e6333b/manager/0.log"
Sep 06 10:56:36.387998 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:36.387973    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c688cbfb6-9qw6m_0d9e9556-7d42-4c39-b689-4e5d9dd51711/copy-manifests/0.log"
Sep 06 10:56:36.398588 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:36.398564    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c688cbfb6-9qw6m_0d9e9556-7d42-4c39-b689-4e5d9dd51711/manager/0.log"
Sep 06 10:56:36.997878 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:36.997851    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-58fd75565c-jj787_27a75fd1-ec38-4f57-aabc-448b95e74fb6/maas-api/0.log"
Sep 06 10:56:38.586225 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:38.586196    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-9q6rs_0e222e6f-6645-4bfc-82c7-36860d344b20/manager/0.log"
Sep 06 10:56:38.716414 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:38.716387    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-2n59j_7e758943-97a7-4ea9-bcc1-6eb56cda5615/limitador/0.log"
Sep 06 10:56:39.311631 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:39.311598    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbhc5_b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8/discovery/0.log"
Sep 06 10:56:39.427322 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:39.427290    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f5945d9f-jktb4_1a1ae590-4d8e-443a-9abd-53ebc09b4c25/kube-auth-proxy/0.log"
Sep 06 10:56:39.771868 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:39.771832    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-96c5t_ce415c4a-d418-4975-bc07-d2257bc91f31/payload-pre-processing/0.log"
Sep 06 10:56:40.393754 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:40.393724    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5pfn4_d67fdca1-851d-42b3-bcba-3ebe5ba27351/main/0.log"
Sep 06 10:56:40.514806 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:40.514775    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-zbx4l_9eefec55-0993-4ba7-ac4b-33731e9553d9/main/0.log"
Sep 06 10:56:40.959965 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:40.959930    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5vgdz_ccf8260b-71ec-48d6-a05a-e0ada8b696f5/main/0.log"
Sep 06 10:56:49.296295 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:49.296264    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ckgmx_431929b2-d956-4151-acea-b8d910e8c6fc/global-pull-secret-syncer/0.log"
Sep 06 10:56:49.457394 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:49.457367    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q7fls_aa7523f5-51a6-4d51-ad6e-afef3666c35d/konnectivity-agent/0.log"
Sep 06 10:56:49.503858 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:49.503835    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-109.ec2.internal_caa56ac7ebc4ca9eb8489a6688c0f5a0/haproxy/0.log"
Sep 06 10:56:54.726590 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:54.726559    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-9q6rs_0e222e6f-6645-4bfc-82c7-36860d344b20/manager/0.log"
Sep 06 10:56:54.754722 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:54.754694    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-2n59j_7e758943-97a7-4ea9-bcc1-6eb56cda5615/limitador/0.log"
Sep 06 10:56:56.574754 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:56.574729    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2x2n2_62085ba7-380b-476e-b255-be186961cedf/node-exporter/0.log"
Sep 06 10:56:56.599564 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:56.599531    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2x2n2_62085ba7-380b-476e-b255-be186961cedf/kube-rbac-proxy/0.log"
Sep 06 10:56:56.625234 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:56.625212    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2x2n2_62085ba7-380b-476e-b255-be186961cedf/init-textfile/0.log"
Sep 06 10:56:57.103705 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:57.103673    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-75nqd_da6ae229-5fa0-401a-b78f-893f261ba735/prometheus-operator/0.log"
Sep 06 10:56:57.151667 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:57.151638    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-75nqd_da6ae229-5fa0-401a-b78f-893f261ba735/kube-rbac-proxy/0.log"
Sep 06 10:56:59.337712 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.337677    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"]
Sep 06 10:56:59.338082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.337995    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5f7374-5300-400d-adc7-f47a19046797" containerName="maas-api"
Sep 06 10:56:59.338082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.338006    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5f7374-5300-400d-adc7-f47a19046797" containerName="maas-api"
Sep 06 10:56:59.338082 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.338068    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5f7374-5300-400d-adc7-f47a19046797" containerName="maas-api"
Sep 06 10:56:59.340802 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.340783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.342888 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.342857    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qdn88\"/\"kube-root-ca.crt\""
Sep 06 10:56:59.343303 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.343285    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qdn88\"/\"openshift-service-ca.crt\""
Sep 06 10:56:59.343398 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.343326    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qdn88\"/\"default-dockercfg-lbxvv\""
Sep 06 10:56:59.348003 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.347979    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"]
Sep 06 10:56:59.501937 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.501901    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-proc\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.501937 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.501940    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-sys\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.502144 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.501984    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-podres\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.502144 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.502001    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmf2\" (UniqueName: \"kubernetes.io/projected/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-kube-api-access-jrmf2\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.502144 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.502029    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-lib-modules\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.602885 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.602805    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-proc\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.602885 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.602846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-sys\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.602885 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.602881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-podres\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.603084 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.602899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmf2\" (UniqueName: \"kubernetes.io/projected/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-kube-api-access-jrmf2\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.603084 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.602926    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-proc\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.603084 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.602928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-lib-modules\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.603084 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.603006    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-sys\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.603084 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.603021    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-podres\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.603084 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.603024    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-lib-modules\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.610074 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.610049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmf2\" (UniqueName: \"kubernetes.io/projected/6a7b7eab-3a26-4e81-84b5-8b85441a1d32-kube-api-access-jrmf2\") pod \"perf-node-gather-daemonset-nz25w\" (UID: \"6a7b7eab-3a26-4e81-84b5-8b85441a1d32\") " pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.651873 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.651852    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:56:59.922596 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:56:59.922570    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"]
Sep 06 10:56:59.924215 ip-10-0-130-109 kubenswrapper[2583]: W0906 10:56:59.924182    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a7b7eab_3a26_4e81_84b5_8b85441a1d32.slice/crio-9b8d0341e91bd4832bb935287a426018f8a97a7f9cdaf3b47671a624e1c33220 WatchSource:0}: Error finding container 9b8d0341e91bd4832bb935287a426018f8a97a7f9cdaf3b47671a624e1c33220: Status 404 returned error can't find the container with id 9b8d0341e91bd4832bb935287a426018f8a97a7f9cdaf3b47671a624e1c33220
Sep 06 10:57:00.120062 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:00.120026    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w" event={"ID":"6a7b7eab-3a26-4e81-84b5-8b85441a1d32","Type":"ContainerStarted","Data":"0c71073f32cb354465fd1d09a11b1dff8730e000b5c951a0e02177cfd688702f"}
Sep 06 10:57:00.120225 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:00.120078    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w" event={"ID":"6a7b7eab-3a26-4e81-84b5-8b85441a1d32","Type":"ContainerStarted","Data":"9b8d0341e91bd4832bb935287a426018f8a97a7f9cdaf3b47671a624e1c33220"}
Sep 06 10:57:00.120225 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:00.120130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:57:00.135279 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:00.135233    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w" podStartSLOduration=1.135220444 podStartE2EDuration="1.135220444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:57:00.134945885 +0000 UTC m=+2111.407587388" watchObservedRunningTime="2026-09-06 10:57:00.135220444 +0000 UTC m=+2111.407861947"
Sep 06 10:57:01.673968 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:01.673939    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2kcjx_ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1/dns/0.log"
Sep 06 10:57:01.695110 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:01.695083    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2kcjx_ef2b5a90-8d9e-4821-a415-f2f0ffc7c8f1/kube-rbac-proxy/0.log"
Sep 06 10:57:01.820443 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:01.820413    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jcbhz_15b6b3de-8170-431a-b270-81ebb6c5f7e9/dns-node-resolver/0.log"
Sep 06 10:57:02.443708 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:02.443680    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vd727_9b4c8743-d239-4ff4-97d3-55e1e861cc1e/node-ca/0.log"
Sep 06 10:57:03.713563 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:03.713528    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbhc5_b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8/discovery/0.log"
Sep 06 10:57:03.733356 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:03.733313    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f5945d9f-jktb4_1a1ae590-4d8e-443a-9abd-53ebc09b4c25/kube-auth-proxy/0.log"
Sep 06 10:57:03.836789 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:03.836762    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-96c5t_ce415c4a-d418-4975-bc07-d2257bc91f31/payload-pre-processing/0.log"
Sep 06 10:57:04.799527 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:04.799499    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nb6zp_ecaebe8d-1299-48ba-80d5-78378304e826/serve-healthcheck-canary/0.log"
Sep 06 10:57:05.351168 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:05.351135    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48hq5_3c438715-ccf5-4929-b41d-501ead22ee0a/kube-rbac-proxy/0.log"
Sep 06 10:57:05.370228 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:05.370202    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48hq5_3c438715-ccf5-4929-b41d-501ead22ee0a/exporter/0.log"
Sep 06 10:57:05.389627 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:05.389606    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-48hq5_3c438715-ccf5-4929-b41d-501ead22ee0a/extractor/0.log"
Sep 06 10:57:06.134106 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:06.134081    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qdn88/perf-node-gather-daemonset-nz25w"
Sep 06 10:57:07.950815 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:07.950781    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6ff9479bc9-hcnng_2f2726ea-8703-407e-97f9-333fc0ceb397/manager/0.log"
Sep 06 10:57:07.969283 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:07.969260    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-nxqbr_474c1520-fc8c-4049-86f3-4d2331b70da1/server/0.log"
Sep 06 10:57:08.050552 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:08.050532    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8vnx9_46181b28-9d0e-4374-96a9-7df450e6333b/manager/0.log"
Sep 06 10:57:08.077836 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:08.077812    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c688cbfb6-9qw6m_0d9e9556-7d42-4c39-b689-4e5d9dd51711/manager/0.log"
Sep 06 10:57:08.097052 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:08.097032    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c688cbfb6-9qw6m_0d9e9556-7d42-4c39-b689-4e5d9dd51711/copy-manifests/0.log"
Sep 06 10:57:09.726800 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:09.726774    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lv6kk_e89f0c1c-04f3-4f2a-8676-7b23ccca2c97/openshift-lws-operator/0.log"
Sep 06 10:57:10.536893 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:10.536809    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbhc5_b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8/discovery/0.log"
Sep 06 10:57:10.546844 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:10.546817    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f5945d9f-jktb4_1a1ae590-4d8e-443a-9abd-53ebc09b4c25/kube-auth-proxy/0.log"
Sep 06 10:57:10.623596 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:10.623563    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-96c5t_ce415c4a-d418-4975-bc07-d2257bc91f31/payload-pre-processing/0.log"
Sep 06 10:57:12.404438 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:12.404395    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-nbhc5_b52301ae-87ac-4c12-9b4e-9f8c4d3b90f8/discovery/0.log"
Sep 06 10:57:12.420827 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:12.420777    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68f5945d9f-jktb4_1a1ae590-4d8e-443a-9abd-53ebc09b4c25/kube-auth-proxy/0.log"
Sep 06 10:57:12.503645 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:12.503608    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-96c5t_ce415c4a-d418-4975-bc07-d2257bc91f31/payload-pre-processing/0.log"
Sep 06 10:57:16.105211 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:16.105174    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9c9ns_f4a955c6-baeb-4527-afc5-0bb8166c0f75/kube-multus/0.log"
Sep 06 10:57:16.128920 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:16.128896    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5lcn9_56811c57-413e-474d-9744-5827780e111d/kube-multus-additional-cni-plugins/0.log"
Sep 06 10:57:16.151860 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:16.151838    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5lcn9_56811c57-413e-474d-9744-5827780e111d/egress-router-binary-copy/0.log"
Sep 06 10:57:16.171778 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:16.171759    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5lcn9_56811c57-413e-474d-9744-5827780e111d/cni-plugins/0.log"
Sep 06 10:57:16.191835 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:16.191814    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5lcn9_56811c57-413e-474d-9744-5827780e111d/bond-cni-plugin/0.log"
Sep 06 10:57:16.213474 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:16.213451    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5lcn9_56811c57-413e-474d-9744-5827780e111d/routeoverride-cni/0.log"
Sep 06 10:57:16.232195 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:16.232174    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5lcn9_56811c57-413e-474d-9744-5827780e111d/whereabouts-cni-bincopy/0.log"
Sep 06 10:57:16.252682 ip-10-0-130-109 kubenswrapper[2583]: I0906 10:57:16.252653    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5lcn9_56811c57-413e-474d-9744-5827780e111d/whereabouts-cni/0.log"